Note: This is test shard 4 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/1, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> >
[ RUN ] AllTypesItest/1.TestTimestampPadding
2026-08-10T13:58:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-10T13:58:06Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 13:58:06.614358 15930 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasksE71nF/build/debug/bin/kudu
/tmp/dist-test-tasksE71nF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.142.190:39225
--webserver_interface=127.15.142.190
--webserver_port=0
--builtin_ntp_servers=127.15.142.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.142.190:39225 with env {}
W20260810 13:58:06.723351 15946 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 13:58:06.723606 15946 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 13:58:06.723645 15946 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260810 13:58:06.727134 15946 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 13:58:06.727216 15946 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260810 13:58:06.727236 15946 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 13:58:06.727255 15946 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260810 13:58:06.731405 15946 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.142.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.142.190:39225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.142.190:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.142.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 10 Aug 2026 13:43:13 UTC on bdcb31816ec0
build id 13272
I20260810 13:58:06.732646 15946 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 13:58:06.733934 15946 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260810 13:58:06.739820 15957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:06.739763 15958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:06.739758 15961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 13:58:06.740154 15946 server_base.cc:1061] running on GCE node
I20260810 13:58:06.740651 15946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 13:58:06.741670 15946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 13:58:06.742836 15946 hybrid_clock.cc:648] HybridClock initialized: now 1786370286742815 us; error 109 us; skew 500 ppm
I20260810 13:58:06.744621 15946 webserver.cc:533] Webserver started at http://127.15.142.190:37199/ using document root <none> and password file <none>
I20260810 13:58:06.745164 15946 fs_manager.cc:362] Metadata directory not provided
I20260810 13:58:06.745218 15946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 13:58:06.745455 15946 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 13:58:06.747046 15946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/instance:
uuid: "e403a6cc5830489399535314c3a0e1cf"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.747527 15946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal/instance:
uuid: "e403a6cc5830489399535314c3a0e1cf"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.750995 15946 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 13:58:06.753175 15967 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:06.754254 15946 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 13:58:06.754400 15946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data,/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal
uuid: "e403a6cc5830489399535314c3a0e1cf"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.754526 15946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 13:58:06.772673 15946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260810 13:58:06.773447 15946 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 13:58:06.773663 15946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 13:58:06.781910 15946 rpc_server.cc:307] RPC server started. Bound to: 127.15.142.190:39225
I20260810 13:58:06.781926 16035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.142.190:39225 every 8 connection(s)
I20260810 13:58:06.782998 15946 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb
I20260810 13:58:06.785847 16036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:06.790948 16036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: Bootstrap starting.
I20260810 13:58:06.790977 15930 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasksE71nF/build/debug/bin/kudu as pid 15946
I20260810 13:58:06.791136 15930 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/wal/instance
I20260810 13:58:06.793352 16036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:06.794298 16036 log.cc:826] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: Log is configured to *not* fsync() on all Append() calls
I20260810 13:58:06.796440 16036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: No bootstrap required, opened a new log
I20260810 13:58:06.799815 16036 raft_consensus.cc:359] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } }
I20260810 13:58:06.800033 16036 raft_consensus.cc:385] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:06.800128 16036 raft_consensus.cc:740] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e403a6cc5830489399535314c3a0e1cf, State: Initialized, Role: FOLLOWER
I20260810 13:58:06.800559 16036 consensus_queue.cc:260] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } }
I20260810 13:58:06.800684 16036 raft_consensus.cc:399] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 13:58:06.800766 16036 raft_consensus.cc:493] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 13:58:06.800884 16036 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:06.801805 16036 raft_consensus.cc:515] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } }
I20260810 13:58:06.802155 16036 leader_election.cc:304] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e403a6cc5830489399535314c3a0e1cf; no voters:
I20260810 13:58:06.802505 16036 leader_election.cc:290] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 13:58:06.802613 16047 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 13:58:06.802783 16047 raft_consensus.cc:697] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [term 1 LEADER]: Becoming Leader. State: Replica: e403a6cc5830489399535314c3a0e1cf, State: Running, Role: LEADER
I20260810 13:58:06.803138 16047 consensus_queue.cc:237] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } }
I20260810 13:58:06.803737 16036 sys_catalog.cc:565] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 13:58:06.805006 16050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e403a6cc5830489399535314c3a0e1cf. Latest consensus state: current_term: 1 leader_uuid: "e403a6cc5830489399535314c3a0e1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } } }
I20260810 13:58:06.805058 16048 sys_catalog.cc:455] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e403a6cc5830489399535314c3a0e1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e403a6cc5830489399535314c3a0e1cf" member_type: VOTER last_known_addr { host: "127.15.142.190" port: 39225 } } }
I20260810 13:58:06.805146 16050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [sys.catalog]: This master's current role is: LEADER
I20260810 13:58:06.805161 16048 sys_catalog.cc:458] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf [sys.catalog]: This master's current role is: LEADER
I20260810 13:58:06.806218 16063 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260810 13:58:06.809574 16063 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260810 13:58:06.814909 16063 catalog_manager.cc:1383] Generated new cluster ID: 5e8a5c4387bb456eaa0600f51c1f8a56
I20260810 13:58:06.814991 16063 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260810 13:58:06.830294 16063 catalog_manager.cc:1406] Generated new certificate authority record
I20260810 13:58:06.831269 16063 catalog_manager.cc:1540] Loading token signing keys...
I20260810 13:58:06.838826 16063 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: Generated new TSK 0
I20260810 13:58:06.839507 16063 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260810 13:58:06.846410 15930 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasksE71nF/build/debug/bin/kudu
/tmp/dist-test-tasksE71nF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.142.129:0
--local_ip_for_outbound_sockets=127.15.142.129
--webserver_interface=127.15.142.129
--webserver_port=0
--tserver_master_addrs=127.15.142.190:39225
--builtin_ntp_servers=127.15.142.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260810 13:58:06.961673 16074 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 13:58:06.961899 16074 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 13:58:06.961984 16074 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260810 13:58:06.965430 16074 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260810 13:58:06.965517 16074 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 13:58:06.965536 16074 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260810 13:58:06.965627 16074 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.142.129
I20260810 13:58:06.969717 16074 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.142.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.142.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.142.129
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.142.190:39225
--never_fsync=true
--heap_profile_path=/tmp/kudu.16074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.142.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 10 Aug 2026 13:43:13 UTC on bdcb31816ec0
build id 13272
I20260810 13:58:06.970855 16074 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 13:58:06.972047 16074 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260810 13:58:06.978401 16082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:06.978446 16083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:06.978724 16085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 13:58:06.979112 16074 server_base.cc:1061] running on GCE node
I20260810 13:58:06.979548 16074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 13:58:06.980193 16074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 13:58:06.981411 16074 hybrid_clock.cc:648] HybridClock initialized: now 1786370286981395 us; error 32 us; skew 500 ppm
I20260810 13:58:06.983357 16074 webserver.cc:533] Webserver started at http://127.15.142.129:44795/ using document root <none> and password file <none>
I20260810 13:58:06.983960 16074 fs_manager.cc:362] Metadata directory not provided
I20260810 13:58:06.984058 16074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 13:58:06.984272 16074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 13:58:06.985996 16074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/instance:
uuid: "8719258dfba14022bdd6f0e728e3d687"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.986502 16074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal/instance:
uuid: "8719258dfba14022bdd6f0e728e3d687"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.989933 16074 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 13:58:06.992249 16094 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:06.993244 16074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 13:58:06.993425 16074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal
uuid: "8719258dfba14022bdd6f0e728e3d687"
format_stamp: "Formatted at 2026-08-10 13:58:06 on dist-test-slave-bw88"
I20260810 13:58:06.993549 16074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 13:58:07.007553 16074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260810 13:58:07.008255 16074 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 13:58:07.008474 16074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 13:58:07.009100 16074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260810 13:58:07.010289 16074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 13:58:07.010371 16074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.010442 16074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260810 13:58:07.010490 16074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.020877 16074 rpc_server.cc:307] RPC server started. Bound to: 127.15.142.129:45039
I20260810 13:58:07.020890 16254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.142.129:45039 every 8 connection(s)
I20260810 13:58:07.022068 16074 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb
I20260810 13:58:07.024348 15930 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasksE71nF/build/debug/bin/kudu as pid 16074
I20260810 13:58:07.024461 15930 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/wal/instance
I20260810 13:58:07.026996 15930 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasksE71nF/build/debug/bin/kudu
/tmp/dist-test-tasksE71nF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.142.130:0
--local_ip_for_outbound_sockets=127.15.142.130
--webserver_interface=127.15.142.130
--webserver_port=0
--tserver_master_addrs=127.15.142.190:39225
--builtin_ntp_servers=127.15.142.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 13:58:07.032824 16256 heartbeater.cc:344] Connected to a master server at 127.15.142.190:39225
I20260810 13:58:07.033131 16256 heartbeater.cc:461] Registering TS with master...
I20260810 13:58:07.033787 16256 heartbeater.cc:507] Master 127.15.142.190:39225 requested a full tablet report, sending...
I20260810 13:58:07.035765 15990 ts_manager.cc:194] Registered new tserver with Master: 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039)
I20260810 13:58:07.037626 15990 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.15.142.129:41809
W20260810 13:58:07.145982 16260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 13:58:07.146240 16260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 13:58:07.146292 16260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260810 13:58:07.149943 16260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260810 13:58:07.150049 16260 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 13:58:07.150067 16260 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260810 13:58:07.150115 16260 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.142.130
I20260810 13:58:07.154388 16260 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.142.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.142.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.142.130
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.142.190:39225
--never_fsync=true
--heap_profile_path=/tmp/kudu.16260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.142.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 10 Aug 2026 13:43:13 UTC on bdcb31816ec0
build id 13272
I20260810 13:58:07.155534 16260 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 13:58:07.156668 16260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260810 13:58:07.163825 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:07.163919 16266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:07.163861 16267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 13:58:07.164115 16260 server_base.cc:1061] running on GCE node
I20260810 13:58:07.164631 16260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 13:58:07.165295 16260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 13:58:07.166498 16260 hybrid_clock.cc:648] HybridClock initialized: now 1786370287166476 us; error 40 us; skew 500 ppm
I20260810 13:58:07.168551 16260 webserver.cc:533] Webserver started at http://127.15.142.130:43409/ using document root <none> and password file <none>
I20260810 13:58:07.169190 16260 fs_manager.cc:362] Metadata directory not provided
I20260810 13:58:07.169279 16260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 13:58:07.169575 16260 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 13:58:07.171353 16260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/instance:
uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.171919 16260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal/instance:
uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.175798 16260 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 13:58:07.178346 16282 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.179575 16260 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 13:58:07.179711 16260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal
uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.179832 16260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 13:58:07.196238 16260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260810 13:58:07.196899 16260 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 13:58:07.197058 16260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 13:58:07.197992 16260 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260810 13:58:07.199424 16260 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 13:58:07.199484 16260 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.199536 16260 ts_tablet_manager.cc:616] Registered 0 tablets
I20260810 13:58:07.199597 16260 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.211045 16260 rpc_server.cc:307] RPC server started. Bound to: 127.15.142.130:34809
I20260810 13:58:07.211066 16440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.142.130:34809 every 8 connection(s)
I20260810 13:58:07.212263 16260 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb
I20260810 13:58:07.214526 15930 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasksE71nF/build/debug/bin/kudu as pid 16260
I20260810 13:58:07.214655 15930 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/wal/instance
I20260810 13:58:07.217420 15930 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasksE71nF/build/debug/bin/kudu
/tmp/dist-test-tasksE71nF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.142.131:0
--local_ip_for_outbound_sockets=127.15.142.131
--webserver_interface=127.15.142.131
--webserver_port=0
--tserver_master_addrs=127.15.142.190:39225
--builtin_ntp_servers=127.15.142.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 13:58:07.223181 16441 heartbeater.cc:344] Connected to a master server at 127.15.142.190:39225
I20260810 13:58:07.223452 16441 heartbeater.cc:461] Registering TS with master...
I20260810 13:58:07.224042 16441 heartbeater.cc:507] Master 127.15.142.190:39225 requested a full tablet report, sending...
I20260810 13:58:07.225180 15990 ts_manager.cc:194] Registered new tserver with Master: 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809)
I20260810 13:58:07.225792 15990 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.15.142.130:60193
W20260810 13:58:07.332088 16445 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 13:58:07.332374 16445 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 13:58:07.332484 16445 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260810 13:58:07.336027 16445 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260810 13:58:07.336162 16445 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 13:58:07.336207 16445 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260810 13:58:07.336295 16445 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.142.131
I20260810 13:58:07.340380 16445 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.142.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.142.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.142.131
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.142.190:39225
--never_fsync=true
--heap_profile_path=/tmp/kudu.16445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.142.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 10 Aug 2026 13:43:13 UTC on bdcb31816ec0
build id 13272
I20260810 13:58:07.341642 16445 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 13:58:07.342861 16445 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260810 13:58:07.349699 16454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:07.349717 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 13:58:07.349712 16459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 13:58:07.349861 16445 server_base.cc:1061] running on GCE node
I20260810 13:58:07.350515 16445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 13:58:07.351079 16445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 13:58:07.352272 16445 hybrid_clock.cc:648] HybridClock initialized: now 1786370287352260 us; error 35 us; skew 500 ppm
I20260810 13:58:07.354154 16445 webserver.cc:533] Webserver started at http://127.15.142.131:42821/ using document root <none> and password file <none>
I20260810 13:58:07.354750 16445 fs_manager.cc:362] Metadata directory not provided
I20260810 13:58:07.354833 16445 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 13:58:07.355069 16445 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 13:58:07.356714 16445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/instance:
uuid: "8fa5636ed9f94804ae2315839b0c5443"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.357223 16445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal/instance:
uuid: "8fa5636ed9f94804ae2315839b0c5443"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.360690 16445 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 13:58:07.363217 16468 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.364342 16445 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 13:58:07.364504 16445 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal
uuid: "8fa5636ed9f94804ae2315839b0c5443"
format_stamp: "Formatted at 2026-08-10 13:58:07 on dist-test-slave-bw88"
I20260810 13:58:07.364620 16445 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 13:58:07.376892 16445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260810 13:58:07.377586 16445 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 13:58:07.377784 16445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 13:58:07.378369 16445 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260810 13:58:07.379460 16445 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 13:58:07.379536 16445 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.379608 16445 ts_tablet_manager.cc:616] Registered 0 tablets
I20260810 13:58:07.379658 16445 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 13:58:07.390293 16445 rpc_server.cc:307] RPC server started. Bound to: 127.15.142.131:35895
I20260810 13:58:07.390321 16641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.142.131:35895 every 8 connection(s)
I20260810 13:58:07.391309 16445 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/info.pb
I20260810 13:58:07.395305 15930 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasksE71nF/build/debug/bin/kudu as pid 16445
I20260810 13:58:07.395462 15930 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/wal/instance
I20260810 13:58:07.401165 16642 heartbeater.cc:344] Connected to a master server at 127.15.142.190:39225
I20260810 13:58:07.401501 16642 heartbeater.cc:461] Registering TS with master...
I20260810 13:58:07.402082 16642 heartbeater.cc:507] Master 127.15.142.190:39225 requested a full tablet report, sending...
I20260810 13:58:07.403199 15990 ts_manager.cc:194] Registered new tserver with Master: 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:07.403789 15990 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.15.142.131:50915
I20260810 13:58:07.410880 15930 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260810 13:58:07.423230 15990 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42708:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT16
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252*\004\001\000\000\000TU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 13:58:07.426378 15990 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 13:58:07.445086 16172 tablet_service.cc:1511] Processing CreateTablet for tablet a36ae99e46dd41c39eb35117232eef67 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 13:58:07.445086 16171 tablet_service.cc:1511] Processing CreateTablet for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 13:58:07.446874 16172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36ae99e46dd41c39eb35117232eef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.445940 16169 tablet_service.cc:1511] Processing CreateTablet for tablet f4b652756bc44c3e9000c0982a04017f (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 13:58:07.447603 16169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b652756bc44c3e9000c0982a04017f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.447250 16532 tablet_service.cc:1511] Processing CreateTablet for tablet f4b652756bc44c3e9000c0982a04017f (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 13:58:07.447359 16339 tablet_service.cc:1511] Processing CreateTablet for tablet f4b652756bc44c3e9000c0982a04017f (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 21844 <= VALUES
I20260810 13:58:07.448426 16532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b652756bc44c3e9000c0982a04017f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.448550 16339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b652756bc44c3e9000c0982a04017f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.447359 16342 tablet_service.cc:1511] Processing CreateTablet for tablet a36ae99e46dd41c39eb35117232eef67 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 13:58:07.449186 16342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36ae99e46dd41c39eb35117232eef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.449774 16341 tablet_service.cc:1511] Processing CreateTablet for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 13:58:07.450042 16341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.453620 16533 tablet_service.cc:1511] Processing CreateTablet for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION 10922 <= VALUES < 21844
I20260810 13:58:07.453965 16533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.454530 16171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.455111 16534 tablet_service.cc:1511] Processing CreateTablet for tablet a36ae99e46dd41c39eb35117232eef67 (DEFAULT_TABLE table=all-types-table [id=dca50103d67841c8a59ca5b349f1d538]), partition=RANGE (key) PARTITION VALUES < 10922
I20260810 13:58:07.455413 16534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36ae99e46dd41c39eb35117232eef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 13:58:07.457477 16666 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: Bootstrap starting.
I20260810 13:58:07.457779 16669 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Bootstrap starting.
I20260810 13:58:07.458146 16668 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: Bootstrap starting.
I20260810 13:58:07.459859 16666 tablet_bootstrap.cc:654] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.459918 16669 tablet_bootstrap.cc:654] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.460227 16668 tablet_bootstrap.cc:654] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.460721 16666 log.cc:826] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: Log is configured to *not* fsync() on all Append() calls
I20260810 13:58:07.460791 16669 log.cc:826] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Log is configured to *not* fsync() on all Append() calls
I20260810 13:58:07.461043 16668 log.cc:826] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: Log is configured to *not* fsync() on all Append() calls
I20260810 13:58:07.462801 16669 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: No bootstrap required, opened a new log
I20260810 13:58:07.463029 16666 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: No bootstrap required, opened a new log
I20260810 13:58:07.463029 16668 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: No bootstrap required, opened a new log
I20260810 13:58:07.463042 16669 ts_tablet_manager.cc:1403] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 13:58:07.463225 16668 ts_tablet_manager.cc:1403] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 13:58:07.463224 16666 ts_tablet_manager.cc:1403] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 13:58:07.466078 16669 raft_consensus.cc:359] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.466302 16669 raft_consensus.cc:385] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.466368 16669 raft_consensus.cc:740] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa5636ed9f94804ae2315839b0c5443, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.466701 16666 raft_consensus.cc:359] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.466701 16668 raft_consensus.cc:359] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.466910 16668 raft_consensus.cc:385] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.466943 16666 raft_consensus.cc:385] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.466869 16669 consensus_queue.cc:260] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.467000 16666 raft_consensus.cc:740] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8719258dfba14022bdd6f0e728e3d687, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.467036 16668 raft_consensus.cc:740] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b47106a46ec42ff8aa6ab8ced381ef3, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.467461 16666 consensus_queue.cc:260] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.467458 16668 consensus_queue.cc:260] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.468065 16669 ts_tablet_manager.cc:1434] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 13:58:07.468358 16666 ts_tablet_manager.cc:1434] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 13:58:07.468358 16668 ts_tablet_manager.cc:1434] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 13:58:07.468457 16642 heartbeater.cc:499] Master 127.15.142.190:39225 was elected leader, sending a full tablet report...
I20260810 13:58:07.468487 16669 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443: Bootstrap starting.
I20260810 13:58:07.468708 16441 heartbeater.cc:499] Master 127.15.142.190:39225 was elected leader, sending a full tablet report...
I20260810 13:58:07.468740 16668 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: Bootstrap starting.
I20260810 13:58:07.469805 16669 tablet_bootstrap.cc:654] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.470338 16256 heartbeater.cc:499] Master 127.15.142.190:39225 was elected leader, sending a full tablet report...
I20260810 13:58:07.470569 16668 tablet_bootstrap.cc:654] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.471437 16666 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: Bootstrap starting.
I20260810 13:58:07.472550 16669 tablet_bootstrap.cc:492] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443: No bootstrap required, opened a new log
I20260810 13:58:07.472575 16668 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: No bootstrap required, opened a new log
I20260810 13:58:07.472656 16669 ts_tablet_manager.cc:1403] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 13:58:07.472667 16668 ts_tablet_manager.cc:1403] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 13:58:07.472890 16666 tablet_bootstrap.cc:654] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.473299 16669 raft_consensus.cc:359] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.473286 16668 raft_consensus.cc:359] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.473459 16668 raft_consensus.cc:385] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.473460 16669 raft_consensus.cc:385] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.473505 16669 raft_consensus.cc:740] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa5636ed9f94804ae2315839b0c5443, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.473609 16668 raft_consensus.cc:740] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b47106a46ec42ff8aa6ab8ced381ef3, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.473690 16669 consensus_queue.cc:260] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.473804 16668 consensus_queue.cc:260] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.474179 16669 ts_tablet_manager.cc:1434] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 13:58:07.474395 16669 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443: Bootstrap starting.
I20260810 13:58:07.475310 16668 ts_tablet_manager.cc:1434] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 13:58:07.475564 16668 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3: Bootstrap starting.
I20260810 13:58:07.475627 16666 tablet_bootstrap.cc:492] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: No bootstrap required, opened a new log
I20260810 13:58:07.475720 16666 ts_tablet_manager.cc:1403] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 13:58:07.476341 16666 raft_consensus.cc:359] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.476449 16666 raft_consensus.cc:385] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.476498 16666 raft_consensus.cc:740] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8719258dfba14022bdd6f0e728e3d687, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.476668 16666 consensus_queue.cc:260] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.476812 16668 tablet_bootstrap.cc:654] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.476924 16666 ts_tablet_manager.cc:1434] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 13:58:07.477113 16666 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687: Bootstrap starting.
I20260810 13:58:07.478238 16666 tablet_bootstrap.cc:654] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.478822 16668 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3: No bootstrap required, opened a new log
I20260810 13:58:07.478909 16668 ts_tablet_manager.cc:1403] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 13:58:07.479377 16669 tablet_bootstrap.cc:654] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443: Neither blocks nor log segments found. Creating new log.
I20260810 13:58:07.479458 16668 raft_consensus.cc:359] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.479565 16668 raft_consensus.cc:385] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.479601 16668 raft_consensus.cc:740] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b47106a46ec42ff8aa6ab8ced381ef3, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.479718 16668 consensus_queue.cc:260] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.479856 16666 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687: No bootstrap required, opened a new log
I20260810 13:58:07.479955 16666 ts_tablet_manager.cc:1403] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 13:58:07.480237 16668 ts_tablet_manager.cc:1434] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 13:58:07.480497 16666 raft_consensus.cc:359] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.480599 16666 raft_consensus.cc:385] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.480646 16666 raft_consensus.cc:740] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8719258dfba14022bdd6f0e728e3d687, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.480772 16666 consensus_queue.cc:260] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.481045 16669 tablet_bootstrap.cc:492] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443: No bootstrap required, opened a new log
I20260810 13:58:07.481045 16666 ts_tablet_manager.cc:1434] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 13:58:07.481163 16669 ts_tablet_manager.cc:1403] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 13:58:07.481734 16669 raft_consensus.cc:359] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.481853 16669 raft_consensus.cc:385] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 13:58:07.481889 16669 raft_consensus.cc:740] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa5636ed9f94804ae2315839b0c5443, State: Initialized, Role: FOLLOWER
I20260810 13:58:07.482052 16669 consensus_queue.cc:260] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.482332 16669 ts_tablet_manager.cc:1434] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 13:58:07.483978 16674 raft_consensus.cc:493] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 13:58:07.484157 16674 raft_consensus.cc:515] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.485399 16674 leader_election.cc:290] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809)
I20260810 13:58:07.489823 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" is_pre_election: true
I20260810 13:58:07.490160 16374 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 0.
I20260810 13:58:07.490203 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687" is_pre_election: true
I20260810 13:58:07.490486 16199 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 0.
I20260810 13:58:07.490737 16471 leader_election.cc:304] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8fa5636ed9f94804ae2315839b0c5443; no voters:
I20260810 13:58:07.491048 16674 raft_consensus.cc:2804] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 13:58:07.491125 16674 raft_consensus.cc:493] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 13:58:07.491176 16674 raft_consensus.cc:3060] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.492120 16674 raft_consensus.cc:515] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.492496 16674 leader_election.cc:290] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 election: Requested vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809)
I20260810 13:58:07.492859 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687"
I20260810 13:58:07.493070 16199 raft_consensus.cc:3060] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.493077 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
I20260810 13:58:07.493204 16374 raft_consensus.cc:3060] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.494593 16199 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 1.
I20260810 13:58:07.494710 16374 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 1.
I20260810 13:58:07.494911 16469 leader_election.cc:304] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8719258dfba14022bdd6f0e728e3d687, 8fa5636ed9f94804ae2315839b0c5443; no voters:
I20260810 13:58:07.495139 16674 raft_consensus.cc:2804] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 13:58:07.495391 16674 raft_consensus.cc:697] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa5636ed9f94804ae2315839b0c5443, State: Running, Role: LEADER
I20260810 13:58:07.495746 16674 consensus_queue.cc:237] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.499011 15990 catalog_manager.cc:5719] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131). New cstate: current_term: 1 leader_uuid: "8fa5636ed9f94804ae2315839b0c5443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } health_report { overall_health: HEALTHY } } }
I20260810 13:58:07.520586 16674 raft_consensus.cc:493] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 13:58:07.520753 16674 raft_consensus.cc:515] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.521235 16674 leader_election.cc:290] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809)
I20260810 13:58:07.521803 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" is_pre_election: true
I20260810 13:58:07.521819 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687" is_pre_election: true
I20260810 13:58:07.521983 16374 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 0.
I20260810 13:58:07.522028 16199 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 0.
I20260810 13:58:07.522334 16471 leader_election.cc:304] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8fa5636ed9f94804ae2315839b0c5443; no voters:
I20260810 13:58:07.522549 16674 raft_consensus.cc:2804] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 13:58:07.522624 16674 raft_consensus.cc:493] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 13:58:07.522703 16674 raft_consensus.cc:3060] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Advancing to term 1
W20260810 13:58:07.523317 16257 tablet.cc:2406] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 13:58:07.523631 16674 raft_consensus.cc:515] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.523998 16674 leader_election.cc:290] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 election: Requested vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809)
I20260810 13:58:07.524442 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
I20260810 13:58:07.524446 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "8fa5636ed9f94804ae2315839b0c5443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687"
I20260810 13:58:07.524585 16199 raft_consensus.cc:3060] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.524585 16374 raft_consensus.cc:3060] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.525660 16199 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 1.
I20260810 13:58:07.525660 16374 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa5636ed9f94804ae2315839b0c5443 in term 1.
I20260810 13:58:07.526055 16469 leader_election.cc:304] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8719258dfba14022bdd6f0e728e3d687, 8fa5636ed9f94804ae2315839b0c5443; no voters:
I20260810 13:58:07.526242 16674 raft_consensus.cc:2804] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 13:58:07.526343 16674 raft_consensus.cc:697] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa5636ed9f94804ae2315839b0c5443, State: Running, Role: LEADER
I20260810 13:58:07.526525 16674 consensus_queue.cc:237] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.528394 15990 catalog_manager.cc:5719] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131). New cstate: current_term: 1 leader_uuid: "8fa5636ed9f94804ae2315839b0c5443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } health_report { overall_health: HEALTHY } } }
W20260810 13:58:07.642392 16643 tablet.cc:2406] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 13:58:07.676505 16676 raft_consensus.cc:493] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 13:58:07.676710 16676 raft_consensus.cc:515] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.678013 16676 leader_election.cc:290] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:07.682492 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a36ae99e46dd41c39eb35117232eef67" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" is_pre_election: true
I20260810 13:58:07.682708 16374 raft_consensus.cc:2468] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8719258dfba14022bdd6f0e728e3d687 in term 0.
I20260810 13:58:07.682960 16567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a36ae99e46dd41c39eb35117232eef67" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa5636ed9f94804ae2315839b0c5443" is_pre_election: true
I20260810 13:58:07.683197 16567 raft_consensus.cc:2468] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8719258dfba14022bdd6f0e728e3d687 in term 0.
I20260810 13:58:07.683172 16105 leader_election.cc:304] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters:
I20260810 13:58:07.683490 16676 raft_consensus.cc:2804] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 13:58:07.683738 16676 raft_consensus.cc:493] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 13:58:07.683821 16676 raft_consensus.cc:3060] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.684791 16676 raft_consensus.cc:515] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.685192 16676 leader_election.cc:290] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 1 election: Requested vote from peers 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:07.685743 16567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a36ae99e46dd41c39eb35117232eef67" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa5636ed9f94804ae2315839b0c5443"
I20260810 13:58:07.685768 16374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a36ae99e46dd41c39eb35117232eef67" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3"
I20260810 13:58:07.685863 16567 raft_consensus.cc:3060] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.685913 16374 raft_consensus.cc:3060] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 13:58:07.686744 16374 raft_consensus.cc:2468] T a36ae99e46dd41c39eb35117232eef67 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8719258dfba14022bdd6f0e728e3d687 in term 1.
I20260810 13:58:07.686830 16567 raft_consensus.cc:2468] T a36ae99e46dd41c39eb35117232eef67 P 8fa5636ed9f94804ae2315839b0c5443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8719258dfba14022bdd6f0e728e3d687 in term 1.
I20260810 13:58:07.687129 16105 leader_election.cc:304] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters:
I20260810 13:58:07.687314 16676 raft_consensus.cc:2804] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 13:58:07.687552 16676 raft_consensus.cc:697] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [term 1 LEADER]: Becoming Leader. State: Replica: 8719258dfba14022bdd6f0e728e3d687, State: Running, Role: LEADER
I20260810 13:58:07.687940 16676 consensus_queue.cc:237] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:07.691367 15990 catalog_manager.cc:5719] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129). New cstate: current_term: 1 leader_uuid: "8719258dfba14022bdd6f0e728e3d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } health_report { overall_health: UNKNOWN } } }
W20260810 13:58:07.713577 16442 tablet.cc:2406] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 13:58:07.998764 16674 consensus_queue.cc:1048] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:08.003239 16689 consensus_queue.cc:1048] T f4b652756bc44c3e9000c0982a04017f P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:08.137970 16674 consensus_queue.cc:1048] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:08.141829 16689 consensus_queue.cc:1048] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:08.166085 16694 consensus_queue.cc:1048] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:08.171355 16676 consensus_queue.cc:1048] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 13:58:09.277259 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling FlushMRSOp(a36ae99e46dd41c39eb35117232eef67): perf score=1.000000
I20260810 13:58:09.467502 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling FlushMRSOp(a36ae99e46dd41c39eb35117232eef67): perf score=1.937278
W20260810 13:58:09.473577 16646 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.142.129:45039, user_credentials={real_user=slave}} blocked reactor thread for 52758.9us
I20260810 13:58:09.647045 16643 maintenance_manager.cc:419] P 8fa5636ed9f94804ae2315839b0c5443: Scheduling FlushMRSOp(5f08fd4ce8af4a039a52f3df8cd2f1e5): perf score=1.000000
W20260810 13:58:09.971824 16650 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.142.131:35895, user_credentials={real_user=slave}} blocked reactor thread for 127276us
W20260810 13:58:10.174680 16473 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-2/data/data (496328704 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 13:58:10.188338 16473 tablet_replica_mm_ops.cc:214] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8fa5636ed9f94804ae2315839b0c5443: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 5f08fd4ce8af4a039a52f3df8cd2f1e5's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 13:58:10.188604 16473 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 5f08fd4ce8af4a039a52f3df8cd2f1e5's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f6dc4699d5d google::LogMessage::Fail() at ??:0
@ 0x7f6dc469db23 google::LogMessage::SendToLog() at ??:0
@ 0x7f6dc469975c google::LogMessage::Flush() at ??:0
@ 0x7f6dc469aee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6dc8123b22 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f6dc5bbccf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f6dc5bb9e08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f6dc5bbe139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5617769a6b9c std::function<>::operator()() at ??:0
@ 0x7f6dc5b7a767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6dc5b7b08f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6dc5b7c7db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5617769a6b9c std::function<>::operator()() at ??:0
@ 0x7f6dc5b6ab52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6dc64e86db start_thread at ??:0
@ 0x7f6dc385061f clone at ??:0
W20260810 13:58:10.693928 16650 meta_cache.cc:302] tablet f4b652756bc44c3e9000c0982a04017f: replica 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895) has failed: Network error: recv got EOF from 127.15.142.131:35895 (error 108)
I20260810 13:58:10.709560 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: FlushMRSOp(a36ae99e46dd41c39eb35117232eef67) complete. Timing: real 1.241s user 0.726s sys 0.044s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":5250,"compiler_manager_pool.run_cpu_time_us":341550,"compiler_manager_pool.run_wall_time_us":606808,"dirs.queue_time_us":249,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1143,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3680,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":390,"threads_started":3,"wal-append.queue_time_us":210}
I20260810 13:58:10.710012 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: FlushMRSOp(a36ae99e46dd41c39eb35117232eef67) complete. Timing: real 1.432s user 0.678s sys 0.039s Metrics: {"bytes_written":436310,"cfile_init":1,"compiler_manager_pool.queue_time_us":15895,"compiler_manager_pool.run_cpu_time_us":344184,"compiler_manager_pool.run_wall_time_us":639314,"dirs.queue_time_us":13093,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1098,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3529,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":270208,"thread_start_us":327,"threads_started":3,"wal-append.queue_time_us":310}
I20260810 13:58:10.710994 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling UndoDeltaBlockGCOp(a36ae99e46dd41c39eb35117232eef67): 62922 bytes on disk
I20260810 13:58:10.711176 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling FlushMRSOp(5f08fd4ce8af4a039a52f3df8cd2f1e5): perf score=1.937278
I20260810 13:58:10.711687 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: UndoDeltaBlockGCOp(a36ae99e46dd41c39eb35117232eef67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260810 13:58:10.712232 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling FlushMRSOp(5f08fd4ce8af4a039a52f3df8cd2f1e5): perf score=1.937278
W20260810 13:58:10.794665 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
W20260810 13:58:10.795787 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 13:58:10.838112 16342 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260810 13:58:10.838766 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 0 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:10.870913 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:11.082118 16342 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260810 13:58:11.082106 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 0 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:11.114982 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:11.161631 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: FlushMRSOp(5f08fd4ce8af4a039a52f3df8cd2f1e5) complete. Timing: real 0.449s user 0.429s sys 0.000s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":365,"dirs.run_wall_time_us":4588,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4230,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":231}
I20260810 13:58:11.162438 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling UndoDeltaBlockGCOp(5f08fd4ce8af4a039a52f3df8cd2f1e5): 62922 bytes on disk
I20260810 13:58:11.162930 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: UndoDeltaBlockGCOp(5f08fd4ce8af4a039a52f3df8cd2f1e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20260810 13:58:11.163367 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling FlushMRSOp(f4b652756bc44c3e9000c0982a04017f): perf score=1.937278
I20260810 13:58:11.164323 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: FlushMRSOp(5f08fd4ce8af4a039a52f3df8cd2f1e5) complete. Timing: real 0.453s user 0.435s sys 0.008s Metrics: {"bytes_written":471972,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1245,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2981,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":146,"threads_started":1,"wal-append.queue_time_us":269}
I20260810 13:58:11.164913 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling FlushMRSOp(f4b652756bc44c3e9000c0982a04017f): perf score=1.937278
I20260810 13:58:11.359192 16880 raft_consensus.cc:493] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.359382 16880 raft_consensus.cc:515] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.359917 16880 leader_election.cc:290] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:11.360965 16373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" is_pre_election: true
W20260810 13:58:11.362421 16105 leader_election.cc:336] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.362500 16105 leader_election.cc:304] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8719258dfba14022bdd6f0e728e3d687; no voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8fa5636ed9f94804ae2315839b0c5443
I20260810 13:58:11.362886 16880 raft_consensus.cc:2749] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 13:58:11.406114 16342 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260810 13:58:11.407176 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 0 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable [LEADER]
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:11.433843 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:11.523869 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: FlushMRSOp(f4b652756bc44c3e9000c0982a04017f) complete. Timing: real 0.360s user 0.352s sys 0.000s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2668,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260810 13:58:11.524581 16442 maintenance_manager.cc:419] P 4b47106a46ec42ff8aa6ab8ced381ef3: Scheduling UndoDeltaBlockGCOp(f4b652756bc44c3e9000c0982a04017f): 63419 bytes on disk
I20260810 13:58:11.525043 16288 maintenance_manager.cc:643] P 4b47106a46ec42ff8aa6ab8ced381ef3: UndoDeltaBlockGCOp(f4b652756bc44c3e9000c0982a04017f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20260810 13:58:11.537635 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: FlushMRSOp(f4b652756bc44c3e9000c0982a04017f) complete. Timing: real 0.373s user 0.367s sys 0.002s Metrics: {"bytes_written":472920,"cfile_init":1,"dirs.queue_time_us":84,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1412,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2420,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260810 13:58:11.538321 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling UndoDeltaBlockGCOp(f4b652756bc44c3e9000c0982a04017f): 63419 bytes on disk
I20260810 13:58:11.539014 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: UndoDeltaBlockGCOp(f4b652756bc44c3e9000c0982a04017f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20260810 13:58:11.539466 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling UndoDeltaBlockGCOp(5f08fd4ce8af4a039a52f3df8cd2f1e5): 62922 bytes on disk
I20260810 13:58:11.539834 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: UndoDeltaBlockGCOp(5f08fd4ce8af4a039a52f3df8cd2f1e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20260810 13:58:11.540333 16257 maintenance_manager.cc:419] P 8719258dfba14022bdd6f0e728e3d687: Scheduling UndoDeltaBlockGCOp(a36ae99e46dd41c39eb35117232eef67): 62922 bytes on disk
I20260810 13:58:11.540650 16107 maintenance_manager.cc:643] P 8719258dfba14022bdd6f0e728e3d687: UndoDeltaBlockGCOp(a36ae99e46dd41c39eb35117232eef67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
I20260810 13:58:11.706193 16913 raft_consensus.cc:493] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.706336 16913 raft_consensus.cc:515] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.707634 16913 leader_election.cc:290] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
W20260810 13:58:11.708351 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
W20260810 13:58:11.709318 16286 leader_election.cc:336] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.711284 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687" is_pre_election: true
I20260810 13:58:11.711467 16199 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b47106a46ec42ff8aa6ab8ced381ef3 in term 1.
I20260810 13:58:11.711908 16283 leader_election.cc:304] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters: 8fa5636ed9f94804ae2315839b0c5443
I20260810 13:58:11.712144 16913 raft_consensus.cc:2804] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 13:58:11.712231 16913 raft_consensus.cc:493] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.712309 16913 raft_consensus.cc:3060] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 13:58:11.713295 16913 raft_consensus.cc:515] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.713750 16913 leader_election.cc:290] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: Requested vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:11.714246 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f08fd4ce8af4a039a52f3df8cd2f1e5" candidate_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687"
I20260810 13:58:11.714383 16199 raft_consensus.cc:3060] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Advancing to term 2
W20260810 13:58:11.714845 16286 leader_election.cc:336] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.715438 16199 raft_consensus.cc:2468] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b47106a46ec42ff8aa6ab8ced381ef3 in term 2.
I20260810 13:58:11.715755 16283 leader_election.cc:304] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters: 8fa5636ed9f94804ae2315839b0c5443
I20260810 13:58:11.715942 16913 raft_consensus.cc:2804] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 13:58:11.716154 16913 raft_consensus.cc:697] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 LEADER]: Becoming Leader. State: Replica: 4b47106a46ec42ff8aa6ab8ced381ef3, State: Running, Role: LEADER
I20260810 13:58:11.716502 16913 consensus_queue.cc:237] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.719008 15981 catalog_manager.cc:5719] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 reported cstate change: term changed from 1 to 2, leader changed from 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131) to 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130). New cstate: current_term: 2 leader_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } health_report { overall_health: UNKNOWN } } }
I20260810 13:58:11.743094 16880 raft_consensus.cc:493] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.743263 16880 raft_consensus.cc:515] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.743826 16880 leader_election.cc:290] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:11.744676 16373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "8719258dfba14022bdd6f0e728e3d687" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" is_pre_election: true
W20260810 13:58:11.745653 16105 leader_election.cc:336] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.745747 16105 leader_election.cc:304] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8719258dfba14022bdd6f0e728e3d687; no voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8fa5636ed9f94804ae2315839b0c5443
I20260810 13:58:11.745955 16880 raft_consensus.cc:2749] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 13:58:11.747238 16913 raft_consensus.cc:493] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.747362 16913 raft_consensus.cc:515] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.747782 16913 leader_election.cc:290] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:11.748180 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687" is_pre_election: true
I20260810 13:58:11.748317 16199 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b47106a46ec42ff8aa6ab8ced381ef3 in term 1.
I20260810 13:58:11.748648 16283 leader_election.cc:304] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters:
I20260810 13:58:11.748834 16913 raft_consensus.cc:2804] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 13:58:11.748888 16913 raft_consensus.cc:493] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fa5636ed9f94804ae2315839b0c5443)
I20260810 13:58:11.748939 16913 raft_consensus.cc:3060] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 1 FOLLOWER]: Advancing to term 2
W20260810 13:58:11.748976 16286 leader_election.cc:336] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.749976 16913 raft_consensus.cc:515] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.750312 16913 leader_election.cc:290] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: Requested vote from peers 8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039), 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895)
I20260810 13:58:11.750656 16199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4b652756bc44c3e9000c0982a04017f" candidate_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8719258dfba14022bdd6f0e728e3d687"
I20260810 13:58:11.750792 16199 raft_consensus.cc:3060] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 1 FOLLOWER]: Advancing to term 2
W20260810 13:58:11.751354 16286 leader_election.cc:336] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111)
I20260810 13:58:11.751745 16199 raft_consensus.cc:2468] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b47106a46ec42ff8aa6ab8ced381ef3 in term 2.
I20260810 13:58:11.752080 16283 leader_election.cc:304] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b47106a46ec42ff8aa6ab8ced381ef3, 8719258dfba14022bdd6f0e728e3d687; no voters: 8fa5636ed9f94804ae2315839b0c5443
I20260810 13:58:11.752283 16913 raft_consensus.cc:2804] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 13:58:11.752385 16913 raft_consensus.cc:697] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [term 2 LEADER]: Becoming Leader. State: Replica: 4b47106a46ec42ff8aa6ab8ced381ef3, State: Running, Role: LEADER
I20260810 13:58:11.752554 16913 consensus_queue.cc:237] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } }
I20260810 13:58:11.754266 15981 catalog_manager.cc:5719] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 reported cstate change: term changed from 1 to 2, leader changed from 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131) to 4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130). New cstate: current_term: 2 leader_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b47106a46ec42ff8aa6ab8ced381ef3" member_type: VOTER last_known_addr { host: "127.15.142.130" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fa5636ed9f94804ae2315839b0c5443" member_type: VOTER last_known_addr { host: "127.15.142.131" port: 35895 } health_report { overall_health: UNKNOWN } } }
I20260810 13:58:11.831435 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:11.831456 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:11.852614 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:12.155357 16199 raft_consensus.cc:1275] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 8719258dfba14022bdd6f0e728e3d687 [term 2 FOLLOWER]: Refusing update from remote peer 4b47106a46ec42ff8aa6ab8ced381ef3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 13:58:12.156059 16913 consensus_queue.cc:1048] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260810 13:58:12.160418 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 13:58:12.183650 16199 raft_consensus.cc:1275] T f4b652756bc44c3e9000c0982a04017f P 8719258dfba14022bdd6f0e728e3d687 [term 2 FOLLOWER]: Refusing update from remote peer 4b47106a46ec42ff8aa6ab8ced381ef3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 13:58:12.184295 16913 consensus_queue.cc:1048] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8719258dfba14022bdd6f0e728e3d687" member_type: VOTER last_known_addr { host: "127.15.142.129" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260810 13:58:12.187726 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 13:58:12.413080 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:12.413517 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:12.435539 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:13.248515 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:13.248871 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:13.268388 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:13.371099 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 13:58:14.328120 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:14.328104 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:14.365725 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:14.520249 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260810 13:58:14.557653 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 13:58:15.421298 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:15.421311 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:15.441048 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:15.913753 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260810 13:58:15.914582 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 13:58:16.498205 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:16.498313 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:16.521950 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:16.815182 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 13:58:16.968273 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260810 13:58:17.341236 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 13:58:17.581313 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:17.581418 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:17.607707 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:18.462990 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260810 13:58:18.665853 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:18.665925 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:18.689778 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:19.302733 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260810 13:58:19.656629 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260810 13:58:19.745332 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:19.745638 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:19.767316 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:20.821834 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:20.821954 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:20.848311 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:21.217666 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 13:58:21.218479 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260810 13:58:21.807032 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 13:58:21.901893 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:21.902204 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:21.926688 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:21.975740 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 13:58:21.976563 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 13:58:22.987308 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:22.987306 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:23.010228 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:23.573227 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260810 13:58:23.986737 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 13:58:24.065964 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:24.066226 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:24.086233 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:24.564741 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 13:58:25.141984 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:25.142318 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:25.166759 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:26.141880 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260810 13:58:26.223486 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:26.223460 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:26.248095 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:26.315949 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260810 13:58:26.733808 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:58:26.859560 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260810 13:58:27.238816 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260810 13:58:27.300575 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:27.300854 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:27.325549 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:28.399923 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:28.399948 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:28.424912 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:28.547607 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260810 13:58:28.776217 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260810 13:58:29.444317 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 13:58:29.480808 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:29.480825 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:29.501644 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:30.556512 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:30.556795 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:30.576244 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:31.015578 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260810 13:58:31.277088 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260810 13:58:31.630362 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:31.630401 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:31.651654 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:31.772434 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:58:32.246492 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 13:58:32.247254 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260810 13:58:32.707607 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:32.707608 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:32.734493 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:33.199266 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 13:58:33.791354 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:33.791342 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
W20260810 13:58:33.802297 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:33.817145 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:34.869699 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:34.869810 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:34.889825 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:35.009564 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260810 13:58:35.458280 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260810 13:58:35.944139 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:35.944694 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:35.970643 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:36.596421 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260810 13:58:36.778936 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 13:58:37.024690 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:37.026600 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:37.056530 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:37.515530 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 13:58:37.516367 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260810 13:58:37.764037 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 13:58:38.110760 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:38.110999 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:38.131285 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:39.078514 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 13:58:39.190021 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:39.190749 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:39.211361 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:40.167886 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 13:58:40.267316 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:40.267334 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:40.287570 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:40.501916 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260810 13:58:41.344120 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:41.344219 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:41.366029 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:41.754163 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260810 13:58:42.019338 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 13:58:42.422550 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:42.422796 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:42.449599 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:42.551268 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 13:58:43.009613 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260810 13:58:43.037534 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260810 13:58:43.504244 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:43.505127 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:43.524314 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:43.910547 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260810 13:58:44.588616 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:44.589037 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:44.609784 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:45.366771 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260810 13:58:45.428078 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 13:58:45.698865 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:45.698976 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:45.731242 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:46.165151 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 13:58:46.785547 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:46.786207 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:46.811452 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:47.523998 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:58:47.827821 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 13:58:47.833146 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 13:58:47.866375 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:47.866487 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:47.888010 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:47.989933 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260810 13:58:48.724086 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260810 13:58:48.941372 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:48.941372 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:48.960697 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:50.019574 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:50.019591 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:50.043138 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:50.426403 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260810 13:58:50.529981 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 13:58:50.911499 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260810 13:58:51.101840 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:51.102308 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:51.123070 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:52.178318 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:52.178381 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:52.199399 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:52.859880 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:58:52.860607 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260810 13:58:53.110139 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260810 13:58:53.252512 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:53.252553 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:53.278532 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:53.353463 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 13:58:53.718441 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260810 13:58:54.333081 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:54.333402 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:54.354035 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:55.401072 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260810 13:58:55.412169 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:55.412572 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:55.433857 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:55.758744 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260810 13:58:55.899174 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260810 13:58:56.488255 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:56.488255 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:56.513522 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:57.569543 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:57.570309 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:57.589793 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:58:57.635015 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260810 13:58:58.143687 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:58:58.393772 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:58:58.394718 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260810 13:58:58.485906 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260810 13:58:58.646291 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:58:58.646309 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:58.667972 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:58:59.723717 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:58:59.723747 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:58:59.748018 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:00.207965 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260810 13:59:00.808593 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:00.808969 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:00.833799 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:00.841686 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260810 13:59:01.076256 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260810 13:59:01.887233 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:01.887553 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:01.912557 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:02.467814 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260810 13:59:02.967746 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:02.968940 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:02.988687 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:03.290459 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:59:03.354110 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260810 13:59:03.395388 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:03.396211 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260810 13:59:04.046455 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:04.046892 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:04.073746 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:04.928124 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260810 13:59:05.132845 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:05.132845 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:05.153880 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:05.731925 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260810 13:59:05.961686 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260810 13:59:06.215461 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:06.215461 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:06.239195 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:07.294430 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:07.294430 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:07.318542 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:07.523219 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260810 13:59:08.001013 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 13:59:08.371490 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:08.371510 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:08.392544 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:08.440857 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 13:59:08.658142 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 13:59:08.774991 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260810 13:59:09.447698 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:09.448271 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:09.474224 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:10.143113 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260810 13:59:10.186512 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260810 13:59:10.529563 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:10.529875 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:10.555333 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:11.125559 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 13:59:11.620488 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:11.620528 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:11.647799 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:12.692194 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 13:59:12.704393 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:12.704721 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
W20260810 13:59:12.721902 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:12.724357 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:13.567972 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:13.568893 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260810 13:59:13.688920 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 13:59:13.776386 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:13.777115 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:13.803339 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:14.862048 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:14.863078 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:14.884701 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:15.119072 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260810 13:59:15.345868 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260810 13:59:15.938989 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:15.938992 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:15.960494 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:16.144467 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 13:59:17.018836 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:17.018870 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:17.041712 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:17.918303 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260810 13:59:18.098227 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:18.098284 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:18.119063 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:18.194121 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260810 13:59:18.467698 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260810 13:59:18.727752 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260810 13:59:18.934927 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260810 13:59:19.175906 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:19.176128 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:19.200721 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:20.256045 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:20.256340 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:20.283087 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:20.598552 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260810 13:59:20.662231 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260810 13:59:20.833483 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260810 13:59:21.339214 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:21.339226 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:21.359689 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:22.421810 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:22.422147 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:22.443766 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:23.121942 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260810 13:59:23.160140 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260810 13:59:23.274190 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260810 13:59:23.499024 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:23.499773 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:23.524224 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:23.812657 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 13:59:23.998028 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260810 13:59:24.581625 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:24.581632 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:24.606989 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:25.478080 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 13:59:25.666539 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:25.666523 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
W20260810 13:59:25.681811 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:25.690580 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:25.963912 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 13:59:26.745482 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:26.745810 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:26.771834 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:27.830276 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:27.832158 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:27.859622 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:27.869194 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260810 13:59:28.218321 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260810 13:59:28.340425 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 13:59:28.913831 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:28.913908 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:28.939141 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:29.068595 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:29.159648 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 13:59:29.993326 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:29.993494 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:30.025660 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:30.180717 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260810 13:59:30.588970 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260810 13:59:30.775362 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260810 13:59:31.082346 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:31.082366 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:31.112243 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:32.172783 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:32.172832 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:32.194970 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:32.457171 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260810 13:59:33.035295 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260810 13:59:33.249059 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 13:59:33.257454 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:33.257449 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:33.278880 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:34.124763 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:34.208863 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260810 13:59:34.332268 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:34.332613 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:34.356716 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:34.974668 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260810 13:59:35.394572 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 13:59:35.410642 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:35.410871 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:35.431874 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:35.771795 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260810 13:59:36.487321 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:36.487330 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:36.512925 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:37.483109 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 13:59:37.570484 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:37.570736 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:37.591116 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:37.981597 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260810 13:59:38.007025 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 13:59:38.649827 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:38.650242 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:38.676051 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:39.495688 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:39.565160 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260810 13:59:39.729929 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:39.730090 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:39.750694 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:40.135807 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260810 13:59:40.475191 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260810 13:59:40.578799 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 13:59:40.805927 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:40.806236 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:40.827087 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:41.886484 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:41.886528 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:41.909057 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:42.508657 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 13:59:42.964123 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:42.964151 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:42.984030 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:43.057806 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260810 13:59:43.274467 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260810 13:59:44.047730 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:44.047966 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:44.070292 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:44.585983 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 13:59:44.852682 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 13:59:44.908982 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260810 13:59:45.130366 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:45.130681 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:45.153481 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:45.589731 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260810 13:59:45.858070 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 13:59:46.209149 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:46.209517 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:46.235587 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:47.247251 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260810 13:59:47.290163 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:47.290241 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:47.313011 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:47.984267 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260810 13:59:48.126654 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260810 13:59:48.368366 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:48.368690 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:48.396688 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 13:59:49.456487 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:49.456488 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:49.483292 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:49.641403 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260810 13:59:49.795161 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:59:50.117869 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 13:59:50.173122 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 13:59:50.540776 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:50.540838 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:50.562352 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:50.646179 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260810 13:59:51.616792 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:51.616995 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:51.638226 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:51.945734 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 13:59:52.693719 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:52.693794 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:52.715692 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:52.806825 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260810 13:59:53.004141 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 13:59:53.770428 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 13:59:53.770468 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:53.791563 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:54.421680 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 13:59:54.855023 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:54.855334 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:54.875700 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:55.240805 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 13:59:55.241734 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260810 13:59:55.360241 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 13:59:55.361027 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 13:59:55.928577 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:55.928730 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:55.947979 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:56.780254 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260810 13:59:57.001398 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:57.001444 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:57.023458 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:57.584270 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260810 13:59:57.892590 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 13:59:58.077255 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:58.077651 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:58.104658 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 13:59:58.945617 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 13:59:59.156524 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 13:59:59.156574 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 13:59:59.182114 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:00:00.240959 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 14:00:00.241616 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:00.261559 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:00.282541 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:00.283384 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260810 14:00:00.388125 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:00.389053 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260810 14:00:01.317286 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 14:00:01.317705 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:01.344901 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:01.524650 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260810 14:00:02.399821 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 14:00:02.400008 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:02.424393 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:02.864660 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260810 14:00:02.939868 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 14:00:03.479902 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260810 14:00:03.479913 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:03.506855 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:04.308956 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260810 14:00:04.561691 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 14:00:04.562103 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:04.584503 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:05.403481 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:00:05.404443 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260810 14:00:05.510939 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:05.511920 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260810 14:00:05.638707 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 14:00:05.638842 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.130:34809, tserver@127.15.142.129:45039
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:05.661927 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:00:06.655320 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20260810 14:00:06.715961 16172 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260810 14:00:06.716423 16342 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e403a6cc5830489399535314c3a0e1cf | 127.15.142.190:39225 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.142.148:44737 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
4b47106a46ec42ff8aa6ab8ced381ef3 | 127.15.142.130:34809 | HEALTHY | <none> | 2 | 0
8719258dfba14022bdd6f0e728e3d687 | 127.15.142.129:45039 | HEALTHY | <none> | 1 | 0
8fa5636ed9f94804ae2315839b0c5443 | 127.15.142.131:35895 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.142.131:35895: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.142.130:34809, 127.15.142.129:45039
ipki_server_key_size | 768 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.129 | experimental | 127.15.142.129:45039
local_ip_for_outbound_sockets | 127.15.142.130 | experimental | 127.15.142.130:34809
never_fsync | true | unsafe,advanced | 127.15.142.130:34809, 127.15.142.129:45039
openssl_security_level_override | 0 | unsafe,hidden | 127.15.142.130:34809, 127.15.142.129:45039
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_format | pb | hidden | 127.15.142.130:34809, 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.142.129:45039
server_dump_info_path | /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.142.130:34809
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.142.130:34809, 127.15.142.129:45039
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.15.142.148:44737 | 127.15.142.130:34809, 127.15.142.129:45039
time_source | builtin | 127.15.142.130:34809, 127.15.142.129:45039
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.15.142.190:39225, tserver@127.15.142.129:45039, tserver@127.15.142.130:34809
Tablet Summary
Tablet a36ae99e46dd41c39eb35117232eef67 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING [LEADER]
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 5f08fd4ce8af4a039a52f3df8cd2f1e5 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet f4b652756bc44c3e9000c0982a04017f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
8719258dfba14022bdd6f0e728e3d687 (127.15.142.129:45039): RUNNING
4b47106a46ec42ff8aa6ab8ced381ef3 (127.15.142.130:34809): RUNNING [LEADER]
8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): TS unavailable
All reported replicas are:
A = 8719258dfba14022bdd6f0e728e3d687
B = 4b47106a46ec42ff8aa6ab8ced381ef3
C = 8fa5636ed9f94804ae2315839b0c5443
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:00:06.737772 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:00:06.961804 15975 maintenance_manager.cc:419] P e403a6cc5830489399535314c3a0e1cf: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033377
W20260810 14:00:06.964233 15972 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-tasksE71nF/test-tmp/all_types-itest.3.AllTypesItest_1.TestTimestampPadding.1786370286586479-15930-0/minicluster-data/master-0/data/data (491442176 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 14:00:06.964864 15972 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P e403a6cc5830489399535314c3a0e1cf: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 14:00:06.964936 15972 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fde57014d5d google::LogMessage::Fail() at ??:0
@ 0x7fde57018b23 google::LogMessage::SendToLog() at ??:0
@ 0x7fde5701475c google::LogMessage::Flush() at ??:0
@ 0x7fde57015ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fde5aa9eb22 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fde58537cf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fde58534e08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7fde58539139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5563fc3a3b9c std::function<>::operator()() at ??:0
@ 0x7fde584f5767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fde584f608f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fde584f77db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5563fc3a3b9c std::function<>::operator()() at ??:0
@ 0x7fde584e5b52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fde58e636db start_thread at ??:0
@ 0x7fde561cb61f clone at ??:0
W20260810 14:00:07.417392 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:00:07.417472 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:00:07.901132 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260810 14:00:07.920599 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 14:00:09.388918 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:00:10.232676 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 14:00:10.417565 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:00:10.608798 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:10.854208 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:00:11.561707 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:00:12.352314 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:00:12.905552 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:00:14.371950 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:00:14.678200 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:00:15.550595 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:00:15.661989 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:16.024984 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:16.843218 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:00:17.289664 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:00:17.904668 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:00:19.309665 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:00:19.677765 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:00:20.519672 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:00:20.694517 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:00:21.138582 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:21.908573 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:00:22.170564 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:00:23.021811 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:00:24.478358 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:00:24.604836 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:00:25.518702 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:00:26.179524 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:26.345963 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:00:27.056212 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:00:27.274761 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:00:27.737954 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:00:29.629669 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:00:29.728883 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:00:30.310048 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:00:31.358160 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:31.427680 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:32.194929 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:00:32.454444 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:00:32.853219 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:00:34.650254 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:00:35.104377 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:00:35.192592 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:00:36.617111 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:00:36.732259 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:00:37.043164 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:00:37.573696 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:00:37.673159 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:00:39.237879 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:00:39.931020 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:00:40.084578 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:00:41.741241 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:41.876406 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:00:41.877214 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:00:42.492843 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:00:42.820263 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:00:44.181517 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:00:45.263866 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:00:45.561620 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:00:46.454283 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:00:46.904887 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:47.076449 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:47.809450 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:00:47.988513 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:00:48.864267 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:00:50.313654 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:00:50.369174 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:00:51.567996 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:00:52.007663 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:00:52.244194 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:00:52.768620 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:00:52.864383 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:00:53.820853 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:00:55.313450 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:00:55.320057 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:00:56.270185 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:00:57.262815 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:00:57.522078 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:00:57.727583 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:00:58.105767 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:00:58.899536 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:01:00.465348 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:01:00.538388 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:01:01.300938 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:01:02.290485 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:01:02.790668 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:02.791424 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:01:03.018918 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:01:03.682276 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:01:05.220108 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:01:05.649003 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:01:06.122589 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:01:07.491384 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (60 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:01:07.491438 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (60 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:01:07.625993 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 14:01:07.654343 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:01:07.842168 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:01:08.050973 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 14:01:08.162966 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:08.619585 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 14:01:10.135180 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:01:10.478554 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:01:11.023855 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:01:12.394659 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:01:12.655398 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:13.058794 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:01:13.417857 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:13.614485 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:01:15.026074 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:01:15.493240 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:01:16.268471 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:01:17.656116 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:01:17.722847 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:01:17.966177 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:01:18.561666 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:01:18.906375 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:01:20.122509 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:01:20.289973 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:01:21.442646 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:01:22.432840 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:01:22.673724 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:01:23.012132 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:23.867841 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:24.067234 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:01:24.870036 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:01:25.227803 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:01:26.430186 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:01:27.185751 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:01:27.746860 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:01:28.259318 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:28.882548 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:01:28.901242 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:01:29.933990 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:01:30.435680 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:01:31.211391 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:01:32.575650 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:01:32.897557 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:01:33.504549 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:33.649684 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:01:34.165690 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:34.747366 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:01:35.410013 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:01:36.180269 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:01:36.950501 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:01:37.928001 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:01:38.509465 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:38.510358 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:01:39.236493 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:39.237439 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:01:40.653563 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:01:40.874130 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:01:41.748217 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:01:42.953042 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:01:43.413435 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:01:43.924950 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:44.057686 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:01:44.544543 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:01:45.319474 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:01:45.878123 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:01:46.801890 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:01:47.862623 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:01:48.687738 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:01:48.932687 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:01:49.344707 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:01:49.889765 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:01:50.410064 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:01:51.374338 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:01:51.884775 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:01:52.852905 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:01:53.666895 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:01:54.191573 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:01:54.435182 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:01:54.928938 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:01:55.244462 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:01:56.239037 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:01:56.916872 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:01:57.746376 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:01:58.708021 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:01:59.270085 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:01:59.506529 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:02:00.118703 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:00.390612 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:02:01.368602 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:02:01.907143 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:02:03.049695 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:02:03.655481 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:02:04.317580 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 14:02:04.661687 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:02:05.160775 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:02:05.720403 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 14:02:06.195993 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 14:02:06.682797 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 14:02:07.565626 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (120 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:02:07.565712 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (120 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:02:08.386370 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 14:02:08.750666 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:02:08.997742 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:02:09.359090 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:02:09.760231 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:02:10.315822 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:10.861544 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:02:11.160475 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:02:11.848667 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:02:13.461791 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:02:13.618140 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:02:14.372807 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:02:15.125630 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:02:15.473929 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:16.166121 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:02:16.235311 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:02:16.792351 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:02:18.703763 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:02:18.819005 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:02:19.231168 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:02:20.172894 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:02:20.602665 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:02:21.122641 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:02:21.307230 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:02:21.626034 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:02:23.552196 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:02:23.894155 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:02:24.346722 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:02:25.273310 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:02:25.607249 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:26.147473 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:02:26.252219 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:02:26.664961 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:02:28.668598 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:02:28.684597 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:02:28.983178 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:02:30.571390 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:02:30.747076 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:02:31.106611 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:02:31.125130 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:02:31.336117 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:02:33.580334 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:02:33.580436 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:02:33.760731 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:02:35.744807 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:02:35.962883 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:36.332784 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:02:36.349012 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:02:36.435051 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:02:38.602084 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:02:38.763736 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:02:39.058265 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:02:41.057931 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:02:41.143859 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:02:41.310478 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:02:41.311473 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:02:41.527024 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:02:43.785068 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:02:43.838938 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:02:44.096565 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:02:46.207234 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:02:46.214480 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:02:46.301795 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:02:46.742148 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:02:46.818701 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:02:48.749763 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:02:48.899459 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:02:49.174047 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:02:51.173816 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:02:51.450011 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:02:51.611161 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:02:51.661662 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:02:51.922016 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:53.683929 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:02:54.108140 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:02:54.234028 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:02:56.207731 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:02:56.658577 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:02:56.686600 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:02:56.687521 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:02:57.104007 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:02:58.929725 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:02:58.935506 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:02:59.152170 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:03:01.423436 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:03:01.569855 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:03:01.668946 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:03:02.035630 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:02.666841 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:04.009801 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:03:04.115667 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:03:04.384893 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:03:06.558388 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:03:06.828212 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 14:03:06.870908 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:03:07.490228 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:07.639593 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (180 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:03:07.639662 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (180 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:03:07.847785 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:03:08.909624 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 14:03:09.442013 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 14:03:09.589820 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:03:10.183238 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
I20260810 14:03:10.559139 22035 consensus_queue.cc:579] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 [LEADER]: Leader has been unable to successfully communicate with peer 8fa5636ed9f94804ae2315839b0c5443 for more than 300 seconds (300.359s)
W20260810 14:03:11.362771 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20260810 14:03:11.770601 22058 consensus_queue.cc:579] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Leader has been unable to successfully communicate with peer 8fa5636ed9f94804ae2315839b0c5443 for more than 300 seconds (300.018s)
W20260810 14:03:11.860432 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20260810 14:03:12.006099 22084 consensus_queue.cc:579] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 [LEADER]: Leader has been unable to successfully communicate with peer 8fa5636ed9f94804ae2315839b0c5443 for more than 300 seconds (300.289s)
W20260810 14:03:12.007817 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 14:03:12.566321 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:12.854374 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:03:13.543870 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 14:03:14.368531 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:03:14.610143 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 14:03:15.626377 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:03:16.769176 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:03:17.150533 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:03:17.574152 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:18.049660 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:03:18.359437 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:19.290530 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:03:19.487169 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:03:20.529315 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:03:21.757457 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:03:21.882409 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:03:22.556710 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:03:22.916064 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 14:03:23.374938 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:03:24.368916 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:03:24.385751 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:03:24.807461 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:03:26.681728 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:03:26.782977 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:03:27.426980 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:03:28.002163 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:28.652982 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:29.096812 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:03:29.261070 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:03:29.977736 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:03:31.674787 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:03:31.789036 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:03:32.560545 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:03:33.369678 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:34.224221 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:34.225075 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:03:34.591169 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:03:35.016206 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:03:36.864118 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:03:37.225525 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:03:37.371743 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:03:38.391570 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:03:39.310474 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:03:39.311347 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:03:39.963413 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:03:40.216353 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:03:41.934667 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:03:42.519066 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:03:42.732589 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:03:43.459630 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:03:44.225977 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:03:44.606385 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:45.141613 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:03:45.321237 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:03:46.493739 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:03:47.619282 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:03:47.696736 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:03:48.636858 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:03:48.874320 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:03:49.700403 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:03:50.206468 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:03:50.490338 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:03:51.277961 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:03:52.734485 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:03:52.881472 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:03:53.795526 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:03:53.806334 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:03:54.862223 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:03:55.179350 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:03:55.444341 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:03:56.555310 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:03:57.514732 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:03:58.027868 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:03:58.938956 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:03:59.039462 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:04:00.004302 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:04:00.037935 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:04:00.461660 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:04:01.420163 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:04:02.656553 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:04:03.032675 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:04:04.023247 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:04:04.052944 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:04:05.173581 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:04:05.190831 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:04:05.562821 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:04:06.804963 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:04:07.510102 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:04:07.693902 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:04:08.079025 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (242 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:04:08.634083 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (241 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:04:09.152379 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:04:09.462337 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:04:09.730114 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:04:10.026014 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:04:10.406006 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:04:11.309912 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:04:11.912777 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 14:04:12.405822 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 14:04:12.690033 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:04:14.157760 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:04:14.343489 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:04:14.995048 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:04:15.222697 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 14:04:15.702353 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:16.844373 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:04:17.908229 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:04:17.912897 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:04:19.160265 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:04:19.198565 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:04:20.135319 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:04:20.374496 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:04:20.875281 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:21.454965 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:04:22.658665 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:04:23.108911 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:04:23.699374 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:04:24.205283 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:04:25.259481 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:04:25.592394 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:04:26.061460 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:26.062227 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:04:27.699446 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:04:28.243721 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:04:28.460867 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:04:29.267287 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:04:30.046156 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:04:30.851035 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:04:30.974349 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:04:31.429963 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:32.553875 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:04:33.405929 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:04:33.441272 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:04:34.611100 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:04:35.076454 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:04:35.658944 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:04:35.889832 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:04:36.450902 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:04:37.763295 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:04:37.944213 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:04:38.752537 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:04:39.862594 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:04:40.402915 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:04:40.667569 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:04:40.905179 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:04:41.801190 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:43.055220 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:04:43.092734 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:04:43.280197 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:04:44.911096 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:04:45.344972 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:04:45.505724 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:04:45.720917 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:04:47.204061 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:47.866582 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:04:48.174667 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:04:48.341288 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:04:50.109812 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:04:50.419374 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:04:50.481135 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:04:50.638000 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:04:52.702006 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:04:52.779495 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:04:53.134750 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:04:53.201036 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:04:54.917583 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:04:55.380728 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:04:55.642643 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:04:55.995050 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:04:57.453920 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:04:57.738365 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:04:58.116300 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:04:58.841794 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:04:59.872977 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:05:00.376736 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:05:00.464972 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:01.542899 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:05:02.646963 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:05:02.892109 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:02.892962 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:05:04.169271 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:05:05.033715 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:05:05.332818 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 14:05:05.516542 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:05:06.580732 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:05:07.492367 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 14:05:07.843353 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:05:08.149235 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (302 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:05:08.359875 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:08.710062 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (301 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:05:09.278716 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:05:09.932925 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:05:10.423640 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:05:10.531306 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:11.816866 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:05:12.444142 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:05:12.760607 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:05:12.906425 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:05:13.472293 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:05:14.622573 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:05:14.805109 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:05:15.304088 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:05:15.697237 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:17.140794 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:05:17.549569 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:05:17.935846 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:05:19.015935 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:19.699286 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:05:19.858768 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:05:20.535408 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:05:20.891428 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:05:22.185111 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:05:22.410630 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:05:23.193624 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:05:24.016355 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:05:24.710068 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:05:24.817467 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:05:25.430313 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:05:26.291975 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:05:27.140515 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:05:27.227747 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:05:27.874131 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:05:29.024987 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:05:29.496964 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:05:29.615064 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:05:30.587611 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:05:31.515420 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:05:31.959128 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:05:32.285881 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:05:32.911185 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:05:34.249744 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:34.879120 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:05:34.895170 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:05:35.138616 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:05:36.746233 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:37.297240 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:05:37.537562 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:05:38.013036 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:05:39.337610 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:05:39.667011 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:05:39.912905 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:05:40.429394 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:05:41.832532 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:05:42.130793 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:05:42.241281 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:05:42.873435 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:05:44.624538 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:05:44.827360 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:44.935534 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:05:45.268319 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:05:47.053946 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:47.247375 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:05:47.674752 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:05:48.078392 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:05:49.701743 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:05:50.225168 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:05:50.281304 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:50.282213 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:05:52.074975 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:05:52.190097 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:05:52.470757 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:05:52.752444 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:05:54.559088 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:05:54.819721 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:05:55.111032 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:05:55.529692 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:05:57.098505 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:05:57.099283 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:05:57.491099 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:05:57.552407 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:05:59.537590 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:06:00.020191 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:06:00.190142 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:06:00.610446 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:06:01.925084 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:06:02.098811 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:02.481273 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:06:02.712445 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:06:04.296602 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:06:04.823423 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:06:05.325850 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:06:05.779028 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:06.560832 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:06:07.084177 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:06:07.158803 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:07.994259 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:06:08.222261 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (362 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:06:08.783018 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (361 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:06:09.339252 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:06:09.764576 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 14:06:10.622999 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:06:10.913141 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:06:11.776232 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:06:12.186347 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:06:12.409049 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 14:06:13.189790 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:06:13.600570 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:06:14.330646 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:06:14.721056 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:06:15.437906 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 14:06:16.427563 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:16.828243 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 14:06:17.193782 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:17.359265 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:06:18.302080 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:06:19.452685 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 14:06:19.709735 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:06:20.818811 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:06:21.777874 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:21.854736 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:06:22.170729 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:06:22.396389 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:23.439568 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:06:24.391896 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:06:24.668291 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:06:25.997027 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:06:26.659119 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:06:26.911967 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:26.912842 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:06:27.672480 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:06:28.665371 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:06:29.164781 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:06:29.335534 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:06:31.313241 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:06:31.541359 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:06:31.851611 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:06:32.462046 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:32.707271 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:06:33.719844 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:06:33.978399 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:06:34.377154 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:06:36.511353 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:06:36.637094 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:06:36.739950 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:06:37.546444 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:37.922348 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:39.041814 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:06:39.062960 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:06:39.065083 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:06:41.393071 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:06:41.393544 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:06:41.658210 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:06:42.839730 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:43.016616 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:43.867265 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:06:44.168499 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:06:44.275475 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:06:46.142417 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:06:46.695042 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:06:47.077126 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:06:48.060650 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:06:48.173947 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:48.562768 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:06:49.153404 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:06:49.618816 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:06:51.029160 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:06:51.675074 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:06:52.065954 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:06:53.165838 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:06:53.574761 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:53.575716 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:06:54.052150 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:06:54.393707 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:06:55.867856 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:06:56.846352 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:06:56.882344 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:06:58.284060 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:06:58.382375 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:06:58.978562 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:06:59.156786 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:06:59.486335 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:07:00.965961 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:07:01.714097 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:07:02.299921 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:07:03.410794 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:07:03.492326 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:07:04.318413 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:07:04.506089 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:04.935937 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:07:05.995334 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 14:07:06.665876 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:07:07.438217 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:07:08.296145 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (422 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:07:08.569991 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:08.573037 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 14:07:08.856931 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (421 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:07:09.232128 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:07:09.894805 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:10.096560 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:07:10.918854 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:07:11.584154 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 14:07:12.494117 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:07:13.483558 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:07:13.663121 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:13.916895 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:07:14.738390 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:07:15.035030 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:07:15.111409 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 14:07:15.989769 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:07:16.559433 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:07:17.583408 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 14:07:18.322979 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:07:18.820732 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:07:19.243239 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:07:20.020179 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:07:20.419129 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:20.973420 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:07:21.657682 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:07:22.568533 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:07:23.517426 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:07:24.104787 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:24.192162 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:07:25.140792 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:07:25.581451 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:07:26.041785 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:07:26.634413 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:07:27.702049 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:07:28.393607 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:07:29.023938 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:07:29.562908 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:07:29.992961 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:07:30.897114 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:30.898061 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:07:31.863420 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:07:32.454057 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:07:33.442178 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:07:34.413983 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:07:34.960626 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:34.961585 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:07:35.687579 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:07:36.180091 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:36.841048 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:07:37.497428 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:07:38.028023 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:07:39.376652 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:07:40.122205 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:40.123015 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:07:40.193584 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:07:41.707324 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260810 14:07:41.858597 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:07:42.765614 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:07:42.888726 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:07:44.605111 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:07:45.215281 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:07:45.425170 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:07:45.426085 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:07:46.778185 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:07:46.900540 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:47.491631 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:07:48.140276 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:07:49.190304 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:07:49.991956 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:07:50.557415 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:07:50.731511 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:07:51.611263 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:07:51.979936 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:07:52.495100 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:07:53.171303 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:07:54.133899 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:07:54.923288 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:07:55.670480 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:07:55.768273 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:07:56.676378 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:07:57.401014 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:07:57.402077 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:07:58.182529 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:07:59.060578 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:08:00.040885 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:08:00.584082 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:08:00.875295 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:08:01.437191 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:08:02.433207 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:02.434048 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:08:02.992691 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:08:03.793437 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:08:04.914665 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 14:08:05.339494 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:08:06.079689 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:08:06.520215 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:08:07.445756 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:07.446537 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 14:08:07.802484 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:08:08.368414 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (482 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:08:08.838650 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:08:08.930656 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (481 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:08:10.066816 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:08:10.310947 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:08:11.190570 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:08:11.191339 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 14:08:12.550721 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:12.551617 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:08:12.733656 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:08:13.656970 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 14:08:14.904510 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:08:15.255820 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:08:15.864099 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:08:16.233596 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:08:16.272619 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:08:17.450609 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:08:17.883304 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 14:08:18.051954 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:08:18.618908 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:08:19.928416 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:08:20.484980 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:08:21.205233 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:08:21.353567 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:08:22.381054 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:08:22.879733 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:08:23.455873 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:08:23.801256 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:08:25.083491 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:08:25.285547 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:08:26.434221 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:08:26.435145 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:08:27.242671 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:08:27.399408 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:08:28.756677 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:08:28.837057 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:08:29.683000 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:08:29.899315 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:08:31.311875 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:08:31.483989 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:08:31.919920 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:08:32.310256 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:08:33.649649 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:08:33.833010 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:34.493537 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:08:34.788069 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:08:36.249020 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:08:36.577363 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:08:37.196954 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:08:37.468271 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:08:38.814056 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:08:38.866149 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:39.653111 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:08:39.807336 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:08:41.454322 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:08:41.600766 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:08:42.091463 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:08:42.187027 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:08:44.248054 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:08:44.345104 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:08:44.595440 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:08:44.764310 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:08:46.614166 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:08:46.688243 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:08:47.197696 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:08:47.372327 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:08:49.222298 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:08:49.293949 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:49.794111 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:08:49.797842 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:08:51.424479 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:08:51.750118 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:08:52.344002 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:08:52.366298 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:08:53.692958 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:08:54.645567 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:08:54.780473 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:08:55.132064 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:08:56.202459 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:08:56.769975 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:08:57.426391 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:08:57.817464 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:08:58.761077 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:08:59.760689 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:08:59.889319 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:00.363948 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:09:01.061589 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:09:01.866227 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:02.390115 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:09:02.887599 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:09:03.637722 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:09:05.087685 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:09:05.377393 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:09:05.378259 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:09:06.207070 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:09:07.092463 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:09:07.659124 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:09:07.795637 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:09:08.443933 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (542 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:09:08.760466 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:09:09.008116 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (541 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:09:10.144353 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:09:10.447764 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:10.448640 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 14:09:11.520907 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:09:12.189687 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:09:12.587409 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:09:13.239434 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 14:09:14.067252 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:09:14.973426 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:09:15.568259 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:15.569087 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:09:16.281986 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:09:16.962631 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:09:17.257011 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:17.451130 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:09:17.817577 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:09:18.656085 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 14:09:19.953261 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 14:09:20.341671 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:09:20.817035 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:09:21.332806 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:09:22.250625 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 14:09:22.305186 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:22.748175 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:09:23.866048 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:09:24.805694 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:09:25.390626 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:09:25.913225 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:26.343638 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:09:27.577890 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:09:27.586668 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:09:27.855964 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:09:29.249150 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:09:29.806223 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:09:30.519579 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:09:30.942312 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:31.856263 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:09:32.350044 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:09:32.827504 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:32.873737 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:09:34.021047 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:09:34.837033 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:09:35.363209 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:09:35.970170 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:36.385483 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:09:37.376756 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:09:37.827807 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:38.003737 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:09:39.015432 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:09:39.728587 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:09:40.519253 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:09:41.413169 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:09:41.565416 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:09:42.420672 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:09:42.836202 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:09:43.014844 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:09:43.962006 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:09:44.878031 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:09:45.474534 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:09:46.421546 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:09:46.468065 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:09:47.547442 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:09:47.868314 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:09:48.076897 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:09:49.201552 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:09:50.095243 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:09:50.372244 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:09:51.521554 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:09:51.957470 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:09:52.668709 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:09:52.904485 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:09:53.107023 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:53.844378 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:09:55.105564 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:09:55.213956 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:09:56.436581 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:09:56.980373 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:09:57.443532 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:09:57.574026 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:09:58.371511 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:09:58.873983 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:09:59.876408 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:10:00.236757 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:10:01.314749 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:10:02.039521 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:02.516952 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:10:02.958590 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:10:03.626945 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:10:03.627799 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:10:04.871340 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:10:05.132346 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:10:05.820523 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:10:07.367956 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:10:07.368937 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:10:07.600304 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:10:07.990384 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:10:08.514367 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (602 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:10:09.048202 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 14:10:09.081316 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (601 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:10:09.790279 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:10:09.849043 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:10:10.525593 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:10:12.310207 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:10:12.612639 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:12.613616 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:10:12.942042 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:10:14.068645 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:10:14.761152 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:10:15.243453 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 14:10:15.626483 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:10:17.054965 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:10:17.948067 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:17.948830 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:10:18.112633 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:10:18.235473 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:10:19.122032 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:10:19.685963 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:10:20.578498 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:10:20.908499 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 14:10:22.346078 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:10:23.091930 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:23.092878 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:10:23.400408 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 14:10:24.341948 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:10:24.657837 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 14:10:25.505991 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:10:25.904891 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:10:27.401808 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 14:10:28.056743 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:10:28.286129 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:10:28.568575 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:10:29.433022 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:10:29.951156 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:10:30.626314 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:10:30.733767 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:10:32.183662 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:10:32.985018 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:10:33.353433 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:10:33.568821 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:34.576884 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:10:34.577780 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:10:35.638818 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:10:35.746057 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:10:37.427183 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:10:37.842818 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:10:38.279340 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:10:39.038064 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:10:39.710969 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:10:40.121017 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:10:40.541402 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:10:40.643026 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:10:42.603210 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:10:42.797595 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:10:43.190811 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:10:44.058655 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:45.056977 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:10:45.187634 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:10:45.566998 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:10:45.648990 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:10:47.671396 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:10:48.035652 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:10:48.292199 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:10:49.424037 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:10:50.193171 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:10:50.233150 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:10:50.236897 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:10:50.795441 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:10:52.606165 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:10:52.734943 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:10:53.017204 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:10:54.497017 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:10:55.018590 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:10:55.439085 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:10:55.439894 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:10:55.455610 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:10:57.587841 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:10:57.977741 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:10:58.051591 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:11:00.048429 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:00.451946 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:11:00.452737 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:11:00.570235 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:11:00.595378 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:11:02.674690 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:11:03.047780 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:11:03.071311 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:11:05.066637 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:11:05.345752 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:05.346484 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:11:05.642311 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:11:05.643139 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:11:07.746320 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:11:07.819919 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:11:08.268711 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:11:08.587123 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (662 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:11:09.153937 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (661 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:11:10.091200 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:11:10.529548 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:11:10.634860 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:10.709419 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:11:10.710251 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:11:12.435897 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 14:11:13.072450 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:11:13.417181 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:11:15.158346 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:11:15.615664 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:11:15.632505 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:11:15.710086 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:11:15.994616 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:11:17.752045 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:11:17.969249 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:11:18.032387 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:11:19.263418 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:11:20.399847 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:11:20.536809 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:11:20.548086 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:11:20.831372 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:11:21.354879 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:11:22.879125 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:11:22.887342 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:11:23.079280 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 14:11:25.339941 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:11:25.515964 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:11:25.652217 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:11:25.865737 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:11:26.647650 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:11:27.654359 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:11:28.018581 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:11:28.233899 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:11:30.375048 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:11:30.593137 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:11:30.729648 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:11:31.315192 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:31.901814 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:11:32.817139 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:11:33.210704 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:11:33.427166 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:11:35.188701 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:11:35.560391 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:11:35.934475 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:11:36.603530 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:36.921247 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:11:37.661667 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:11:37.993427 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:11:38.446444 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:11:40.147797 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:11:40.422726 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:11:40.930357 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:11:41.971961 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:11:42.110261 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:42.621940 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:11:42.764556 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:11:43.742993 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:11:45.027583 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:11:45.182891 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:11:46.515316 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:11:47.017474 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260810 14:11:47.541488 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:47.542273 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:11:47.634217 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:11:48.964105 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:11:50.060091 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:11:50.079406 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:11:51.513200 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:11:52.083737 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:11:52.350265 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:11:52.453739 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:11:52.942340 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:54.139323 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:11:54.968871 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:11:55.030311 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:11:56.867923 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:11:57.254767 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:11:57.338372 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:11:57.480028 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:11:58.287101 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:11:59.353847 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:11:59.807199 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:11:59.901734 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:12:01.975107 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:12:02.177670 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:12:02.412843 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:02.454897 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:12:03.422120 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:04.206389 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:12:04.352494 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:12:04.801079 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:12:06.742012 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:12:06.948107 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:12:07.050014 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:12:07.722671 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 14:12:08.467375 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:12:08.661983 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (722 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:12:09.164937 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:12:09.227151 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (721 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:12:09.386520 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:12:09.627889 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 14:12:11.531247 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:12:11.717737 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:12:12.060746 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:12:12.796608 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:13.513047 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:13.996276 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:12:14.306488 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:12:14.559195 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:12:16.572263 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:12:16.834334 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 14:12:16.861929 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 14:12:18.317441 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:18.685286 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:12:19.193039 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:12:19.196238 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:12:19.511303 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.142.190:39225) | 127.15.142.190:39225 | UNAVAILABLE
Error from 127.15.142.190:39225: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
FAILED
I20260810 14:12:20.384434 15930 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.142.190:39225, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:12:21.328184 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:12:21.660486 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:12:22.007776 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:12:23.479466 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:23.789832 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:23.952056 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 14:12:24.324136 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:12:24.657133 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:12:26.458849 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:12:26.913928 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:12:27.381687 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 14:12:28.761965 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:28.762732 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:12:28.910948 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:29.351888 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 14:12:30.043540 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:12:31.441949 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:12:31.510241 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:12:32.608578 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:12:33.700291 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:12:34.179386 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:12:34.180173 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:12:34.232246 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260810 14:12:35.471977 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:12:36.177976 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:12:36.837733 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
[New LWP 15938]
[New LWP 15939]
[New LWP 15941]
[New LWP 16646]
[New LWP 16648]
[New LWP 16650]
[New LWP 16651]
[New LWP 23616]
[New LWP 23617]
[New LWP 23618]
[New LWP 23620]
[New LWP 23624]
[New LWP 23625]
[New LWP 23627]
[New LWP 23628]
[New LWP 23629]
W20260810 14:12:37.839519 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:12:38.972504 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:12:39.373610 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:12:39.389262 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:12:39.415866 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260810 14:12:40.366083 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 14:12:41.448228 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:12:41.932494 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:12:42.844374 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:12:44.041167 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:12:44.409803 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:44.410648 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:12:44.547690 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:45.201984 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:12:46.469012 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:12:46.958058 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:12:47.609534 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:12:49.094591 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:12:49.595649 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:12:49.681775 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:49.713023 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:12:50.103189 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:12:51.655911 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 14:12:52.183313 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:12:52.330763 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:12:54.594205 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 14:12:54.616416 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260810 14:12:54.617230 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:12:54.818735 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:12:55.195405 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260810 14:12:57.032486 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:12:57.075489 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260810 14:12:57.440550 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:12:59.478353 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260810 14:12:59.631655 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260810 14:12:59.632579 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:12:59.767798 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 14:13:00.298064 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260810 14:13:02.016211 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260810 14:13:02.110028 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 14:13:02.388715 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 14:13:04.243202 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
0x00007f7590609ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f75838eb280 (LWP 15930) "all_types-itest" 0x00007f7590609ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f75818c6700 (LWP 15938) "all_types-itest" 0x00007f759060e32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f757feec700 (LWP 15939) "kernel-watcher-" 0x00007f7590609fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260810 14:13:04.693902 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 20 similar messages]
4 Thread 0x7f757f6eb700 (LWP 15941) "rpc reactor-159" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f757eeea700 (LWP 16646) "rpc reactor-166" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f757e4e2700 (LWP 16648) "rpc reactor-166" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f757dce1700 (LWP 16650) "rpc reactor-166" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f757d4e0700 (LWP 16651) "rpc reactor-166" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f7575cd1700 (LWP 23616) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f757c4de700 (LWP 23617) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f7574ccf700 (LWP 23618) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f757bcdd700 (LWP 23620) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f757acdb700 (LWP 23624) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f757a4da700 (LWP 23625) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f7579cd9700 (LWP 23627) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f75794d8700 (LWP 23628) "rpc reactor-236" 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
W20260810 14:13:04.740948 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
17 Thread 0x7f7578cd7700 (LWP 23629) "client-negotiat" 0x00007f7590609fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260810 14:13:04.744212 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
Thread 17 (Thread 0x7f7578cd7700 (LWP 23629)):
#0 0x00007f7590609fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f758fde121c in kudu::ConditionVariable::WaitFor (this=0x7f7578cd6690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007f758ff0c23d in kudu::ThreadPool::DispatchThread (this=0x5623fdb898c0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:677
#3 0x00007f758ff0d08f in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x5623fe3d7c78) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:815
#4 0x00007f758ff0e7db in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#5 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d7c78) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d7c20) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f75794d8700 (LWP 23628)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc72680, timeout=0.099461202044039965) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc72680, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
W20260810 14:13:05.357546 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 8 similar messages]
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdb80e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdb80e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d6718) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d6718) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d66c0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f7579cd9700 (LWP 23627)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb68680, timeout=0.099470415851101279) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb68680, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdb81898, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdb81890) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d65f8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d65f8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d65a0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f757a4da700 (LWP 23625)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb69700, timeout=0.050758609781041741) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb69700, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fd984498, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fd984490) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d63b8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d63b8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d6360) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f757acdb700 (LWP 23624)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc71600, timeout=0.099408045876771212) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc71600, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fd985898, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fd985890) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d7eb8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d7eb8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d7e60) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f757bcdd700 (LWP 23620)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc71b80, timeout=0.099534832872450352) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc71b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdbc5298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdbc5290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d7018) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d7018) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d6fc0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f7574ccf700 (LWP 23618)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc72c00, timeout=0.099524085875600576) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc72c00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdbc4a98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdbc4a90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d7378) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d7378) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d7320) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f757c4de700 (LWP 23617)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc72100, timeout=0.099511366803199053) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc72100, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdeaf898, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdeaf890) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d6298) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d6298) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d6240) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f7575cd1700 (LWP 23616)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdc70b00, timeout=0.099460739875212312) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdc70b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdeae698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdeae690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fe3d7b58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fe3d7b58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fe3d7b00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f757d4e0700 (LWP 16651)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb67b80, timeout=0.099497510585933924) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb67b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdb80c98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdb80c90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fdba8ef8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fdba8ef8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fdba8ea0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f757dce1700 (LWP 16650)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb67600, timeout=0.099374281242489815) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb67600, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdb80298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdb80290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fdba9018) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fdba9018) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fdba8fc0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f757e4e2700 (LWP 16648)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb66b00, timeout=0.099357085069641471) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb66b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fdb80698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fdb80690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fdba9138) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fdba9138) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fdba90e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f757eeea700 (LWP 16646)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb66580, timeout=0.09930695011280477) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb66580, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fd9b7c98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fd9b7c90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fdba9258) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fdba9258) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fdba9200) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f757f6eb700 (LWP 15941)):
#0 0x00007f758e4f4947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7589e7e25d in epoll_poll (loop=0x5623fdb66000, timeout=0.098993290914222598) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f7589e82ba3 in ev_run (loop=0x5623fdb66000, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f758fef580f in ev::loop_ref::run (this=0x5623fd9b6e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f758bfb40f1 in kudu::rpc::ReactorThread::RunThread (this=0x5623fd9b6e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007f758bfb0bcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5623fda8bd98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007f758bfb88d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fda8bd98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fda8bd40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f757feec700 (LWP 15939)):
#0 0x00007f7590609fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f758fde1033 in kudu::ConditionVariable::WaitUntil (this=0x5623fda8bf38, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:87
#2 0x00007f758fe622f4 in kudu::CountDownLatch::WaitUntil (this=0x5623fda8bf00, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:89
#3 0x00007f758fe6239e in kudu::CountDownLatch::WaitFor (this=0x5623fda8bf00, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:99
#4 0x00007f758fe613b2 in kudu::KernelStackWatchdog::RunThread (this=0x5623fda8be60) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f758fe605c1 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x5623fda8bb58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f758fe61d16 in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005623ca80935a in std::function<void ()>::operator()() const (this=0x5623fda8bb58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007f758fefcb52 in kudu::Thread::SuperviseThread (arg=0x5623fda8bb00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f75818c6700 (LWP 15938)):
#0 0x00007f759060e32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f758fef33a1 in kudu::Subprocess::DoWait (this=0x7f75818c5350, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:849
#2 0x00007f758fef073a in kudu::Subprocess::Wait (this=0x7f75818c5350, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:561
#3 0x00007f758fe55729 in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:260
#4 0x00007f758fe55259 in kudu::PstackWatcher::RunGdbStackDump (pid=15930, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:218
#5 0x00007f758fe54ab7 in kudu::PstackWatcher::DumpPidStacks (pid=15930, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:169
#6 0x00007f758fe54a30 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:162
#7 0x00005623ca807e09 in kudu::<lambda()>::operator()(void) const (__closure=0x5623fda2fff8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:66
#8 0x00005623ca80860a in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9 0x00005623ca80835e in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x00005623ca808d7e in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x5623fda2fff8) at /usr/include/c++/7/thread:234
#11 0x00005623ca808d23 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x5623fda2fff8) at /usr/include/c++/7/thread:243
#12 0x00005623ca808cde in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x5623fda2fff0) at /usr/include/c++/7/thread:186
#13 0x00007f758ee376df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007f75906036db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f758e4f461f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f75838eb280 (LWP 15930)):
#0 0x00007f7590609ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f758fde0ead in kudu::ConditionVariable::Wait (this=0x5623fe3b7db0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:55
W20260810 14:13:06.294611 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#2 0x00007f7591064aa1 in kudu::CountDownLatch::Wait (this=0x5623fe3b7d78) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:79
#3 0x00007f7590e23414 in kudu::Synchronizer::Wait (this=0x7ffd0d537ef0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/async_util.h:58
W20260810 14:13:07.090268 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260810 14:13:07.444720 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1806: this message will repeat every 5th retry.
#4 0x00007f7590bf5c0a in kudu::client::KuduClient::Data::ConnectToCluster (this=0x5623fdc68780, client=0x5623fdbcf960, deadline=..., creds_policy=kudu::rpc::CredentialsPolicy::ANY_CREDENTIALS) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client-internal.cc:828
#5 0x00007f7590bb78a5 in kudu::client::KuduClientBuilder::Build (this=0x7ffd0d538330, client=0x5623fe3d78b0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client.cc:474
W20260810 14:13:08.734988 16441 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (782 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:13:08.749945 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#6 0x00007f7589260a93 in kudu::tools::CreateKuduClient (master_addresses=std::vector of length 1, capacity 1 = {...}, client=0x5623fe3d78b0, can_see_all_replicas=true) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/tool_action_common.cc:773
W20260810 14:13:09.302671 16256 heartbeater.cc:646] Failed to heartbeat to 127.15.142.190:39225 (781 consecutive failures): Network error: Failed to ping master at 127.15.142.190:39225: Client connection negotiation failed: client connection to 127.15.142.190:39225: connect: Connection refused (error 111)
W20260810 14:13:09.451959 16286 consensus_peers.cc:597] T f4b652756bc44c3e9000c0982a04017f P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#7 0x00007f758e2e0ac4 in kudu::tools::RemoteKsckCluster::Connect (this=0x5623fe3d77a0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck_remote.cc:530
#8 0x00007f758e29888e in kudu::tools::Ksck::CheckClusterRunning (this=0x5623fdc09010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:459
#9 0x00007f758e299e84 in kudu::tools::Ksck::Run (this=0x5623fdc09010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:594
W20260810 14:13:09.940740 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20260810 14:13:09.966572 16105 consensus_peers.cc:597] T a36ae99e46dd41c39eb35117232eef67 P 8719258dfba14022bdd6f0e728e3d687 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1811: this message will repeat every 5th retry.
#10 0x00007f758e29c27b in kudu::tools::Ksck::RunAndPrintResults (this=0x5623fdc09010) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:800
#11 0x00007f7591035044 in kudu::ClusterVerifier::RunKsck (this=0x7ffd0d538840) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:113
W20260810 14:13:10.441885 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111) [suppressed 9 similar messages]
#12 0x00007f7591034a03 in kudu::ClusterVerifier::CheckCluster (this=0x7ffd0d538840) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x00005623ca7c4f8b in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x5623fd983900, scanner_setup=..., verifier=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x00005623ca7956c1 in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)3> > >::TestBody (this=0x5623fd983900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:804
#15 0x00007f759002e0ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f759002f82b "the test body", method=<optimized out>, object=0x5623fd983900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x5623fd983900, method=<optimized out>, location=location@entry=0x7f759002f82b "the test body") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f7590022bdb in testing::Test::Run (this=0x5623fd983900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 0x00007f7590022d9d in testing::TestInfo::Run (this=0x5623fd999c20) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#19 0x00007f7590023377 in testing::TestSuite::Run (this=0x5623fd9985a0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#20 0x00007f759002377c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#21 0x00007f759002e60d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f75900313c8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x5623fda5a280) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#22 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5623fda5a280, method=<optimized out>, location=location@entry=0x7f75900313c8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#23 0x00007f7590022e63 in testing::UnitTest::Run (this=0x7f75902421e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#24 0x00005623ca8090d2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#25 0x00005623ca8082bf in main (argc=1, argv=0x7ffd0d538e48) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:115
************************* END STACKS ***************************
F20260810 14:13:10.977314 15938 test_main.cc:69] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786371190 (unix time) try "date -d @1786371190" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003e3a) received by PID 15930 (TID 0x7f75818c6700) from PID 15930; stack trace: ***
@ 0x7f759060e980 (unknown) at ??:0
@ 0x7f758e411e87 gsignal at ??:0
@ 0x7f758e4137f1 abort at ??:0
@ 0x7f758f116d5d google::LogMessage::Fail() at ??:0
@ 0x7f758f11ab23 google::LogMessage::SendToLog() at ??:0
@ 0x7f758f11675c google::LogMessage::Flush() at ??:0
@ 0x7f758f117ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5623ca807f46 _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at ??:0
@ 0x5623ca80860a _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x5623ca80835e _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at ??:0
@ 0x5623ca808d7e _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x5623ca808d23 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at ??:0
@ 0x5623ca808cde _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at ??:0
@ 0x7f758ee376df (unknown) at ??:0
W20260810 14:13:11.006654 16286 consensus_peers.cc:597] T 5f08fd4ce8af4a039a52f3df8cd2f1e5 P 4b47106a46ec42ff8aa6ab8ced381ef3 -> Peer 8fa5636ed9f94804ae2315839b0c5443 (127.15.142.131:35895): Couldn't send request to peer 8fa5636ed9f94804ae2315839b0c5443. Status: Network error: Client connection negotiation failed: client connection to 127.15.142.131:35895: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
@ 0x7f75906036db start_thread at ??:0
@ 0x7f758e4f461f clone at ??:0