Note: This is test shard 5 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/2, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> >
[ RUN ] AllTypesItest/2.TestAllKeyTypes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 14:30:20.923581 25016 test_util.cc:268] Using random seed: 833236571
2026-08-10T14:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:30:20Z Disabled control of system clock
I20260810 14:30:20.956197 25016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTeuddy/build/debug/bin/kudu
/tmp/dist-test-taskTeuddy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.110.62:37293
--webserver_interface=127.24.110.62
--webserver_port=0
--builtin_ntp_servers=127.24.110.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.110.62:37293 with env {}
W20260810 14:30:21.094019 25029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:30:21.094522 25029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:30:21.094604 25029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:30:21.099022 25029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:30:21.099213 25029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:30:21.099290 25029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:30:21.099329 25029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:30:21.105075 25029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.110.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.110.62:37293
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.110.62:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.110.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:30:21.106839 25029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:30:21.109055 25029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:30:21.123030 25036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.123528 25037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.123936 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:30:21.124187 25029 server_base.cc:1034] running on GCE node
I20260810 14:30:21.125015 25029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:30:21.127043 25029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:30:21.128252 25029 hybrid_clock.cc:648] HybridClock initialized: now 1786372221128230 us; error 72 us; skew 500 ppm
I20260810 14:30:21.136056 25029 webserver.cc:460] Webserver started at http://127.24.110.62:46637/ using document root <none> and password file <none>
I20260810 14:30:21.137117 25029 fs_manager.cc:359] Metadata directory not provided
I20260810 14:30:21.137233 25029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:30:21.137740 25029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:30:21.140671 25029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/instance:
uuid: "077af14381a74f028ac4e1ae1fd1fe01"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.141742 25029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal/instance:
uuid: "077af14381a74f028ac4e1ae1fd1fe01"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.147497 25029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:30:21.151453 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.153293 25029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 14:30:21.153558 25029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data,/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal
uuid: "077af14381a74f028ac4e1ae1fd1fe01"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.153726 25029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:30:21.174384 25029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:30:21.175325 25029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:30:21.175585 25029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:30:21.185835 25029 rpc_server.cc:307] RPC server started. Bound to: 127.24.110.62:37293
I20260810 14:30:21.185896 25122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.110.62:37293 every 8 connection(s)
I20260810 14:30:21.187321 25029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb
I20260810 14:30:21.193851 25123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.195029 25016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTeuddy/build/debug/bin/kudu as pid 25029
I20260810 14:30:21.195245 25016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/wal/instance
I20260810 14:30:21.202715 25123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: Bootstrap starting.
I20260810 14:30:21.208142 25123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:21.209589 25123 log.cc:824] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: Log is configured to *not* fsync() on all Append() calls
I20260810 14:30:21.212841 25123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: No bootstrap required, opened a new log
I20260810 14:30:21.219535 25123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } }
I20260810 14:30:21.220031 25123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:21.220191 25123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077af14381a74f028ac4e1ae1fd1fe01, State: Initialized, Role: FOLLOWER
I20260810 14:30:21.221056 25123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } }
I20260810 14:30:21.221307 25123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:30:21.221485 25123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:30:21.221638 25123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:21.223170 25123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } }
I20260810 14:30:21.223917 25123 leader_election.cc:302] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077af14381a74f028ac4e1ae1fd1fe01; no voters:
I20260810 14:30:21.224440 25123 leader_election.cc:288] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:30:21.224725 25128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:30:21.225029 25128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [term 1 LEADER]: Becoming Leader. State: Replica: 077af14381a74f028ac4e1ae1fd1fe01, State: Running, Role: LEADER
I20260810 14:30:21.225662 25128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } }
I20260810 14:30:21.226418 25123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:30:21.228547 25130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077af14381a74f028ac4e1ae1fd1fe01. Latest consensus state: current_term: 1 leader_uuid: "077af14381a74f028ac4e1ae1fd1fe01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } } }
I20260810 14:30:21.228641 25129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077af14381a74f028ac4e1ae1fd1fe01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077af14381a74f028ac4e1ae1fd1fe01" member_type: VOTER last_known_addr { host: "127.24.110.62" port: 37293 } } }
I20260810 14:30:21.228739 25130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [sys.catalog]: This master's current role is: LEADER
I20260810 14:30:21.228760 25129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01 [sys.catalog]: This master's current role is: LEADER
W20260810 14:30:21.234314 25152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:30:21.234472 25152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:30:21.234548 25142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:30:21.235576 25142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:30:21.244145 25142 catalog_manager.cc:1347] Generated new cluster ID: 92664cf08b274163bdab21cd32aef86c
I20260810 14:30:21.244242 25142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:30:21.276419 25142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:30:21.278050 25142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:30:21.285822 25142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: Generated new TSK 0
I20260810 14:30:21.286975 25142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:30:21.291800 25016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTeuddy/build/debug/bin/kudu
/tmp/dist-test-taskTeuddy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.110.1:0
--local_ip_for_outbound_sockets=127.24.110.1
--webserver_interface=127.24.110.1
--webserver_port=0
--tserver_master_addrs=127.24.110.62:37293
--builtin_ntp_servers=127.24.110.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260810 14:30:21.414618 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:30:21.414955 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:30:21.415031 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:30:21.419288 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:30:21.419404 25156 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 14:30:21.419436 25156 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 14:30:21.419510 25156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.110.1
I20260810 14:30:21.424528 25156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.110.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.110.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.110.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.24.110.62:37293
--never_fsync=true
--heap_profile_path=/tmp/kudu.25156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.110.1
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:30:21.425895 25156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:30:21.427691 25156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:30:21.439374 25167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.439383 25163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.439411 25161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:30:21.439697 25156 server_base.cc:1034] running on GCE node
I20260810 14:30:21.440346 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:30:21.441143 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:30:21.442329 25156 hybrid_clock.cc:648] HybridClock initialized: now 1786372221442316 us; error 57 us; skew 500 ppm
I20260810 14:30:21.444640 25156 webserver.cc:460] Webserver started at http://127.24.110.1:38085/ using document root <none> and password file <none>
I20260810 14:30:21.445397 25156 fs_manager.cc:359] Metadata directory not provided
I20260810 14:30:21.445499 25156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:30:21.445778 25156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:30:21.448460 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/instance:
uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.449138 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal/instance:
uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.453713 25156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:30:21.456537 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.458117 25156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:30:21.458279 25156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal
uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.458415 25156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:30:21.474753 25156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:30:21.475592 25156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:30:21.475827 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:30:21.476647 25156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:30:21.478188 25156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:30:21.478286 25156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.478389 25156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:30:21.478438 25156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.489602 25156 rpc_server.cc:307] RPC server started. Bound to: 127.24.110.1:43075
I20260810 14:30:21.489629 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.110.1:43075 every 8 connection(s)
I20260810 14:30:21.490763 25156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb
I20260810 14:30:21.501089 25016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTeuddy/build/debug/bin/kudu as pid 25156
I20260810 14:30:21.501257 25016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/wal/instance
I20260810 14:30:21.502894 25343 heartbeater.cc:347] Connected to a master server at 127.24.110.62:37293
I20260810 14:30:21.503181 25343 heartbeater.cc:464] Registering TS with master...
I20260810 14:30:21.503772 25343 heartbeater.cc:511] Master 127.24.110.62:37293 requested a full tablet report, sending...
I20260810 14:30:21.503798 25016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTeuddy/build/debug/bin/kudu
/tmp/dist-test-taskTeuddy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.110.2:0
--local_ip_for_outbound_sockets=127.24.110.2
--webserver_interface=127.24.110.2
--webserver_port=0
--tserver_master_addrs=127.24.110.62:37293
--builtin_ntp_servers=127.24.110.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 14:30:21.506196 25075 ts_manager.cc:194] Registered new tserver with Master: 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075)
I20260810 14:30:21.507959 25075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.110.1:56265
W20260810 14:30:21.628698 25348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:30:21.629016 25348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:30:21.629099 25348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:30:21.633047 25348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:30:21.633116 25348 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 14:30:21.633132 25348 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 14:30:21.633208 25348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.110.2
I20260810 14:30:21.638127 25348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.110.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.110.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.110.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.24.110.62:37293
--never_fsync=true
--heap_profile_path=/tmp/kudu.25348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.110.2
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:30:21.639297 25348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:30:21.640750 25348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:30:21.650910 25356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.650898 25355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.650898 25358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:30:21.651224 25348 server_base.cc:1034] running on GCE node
I20260810 14:30:21.651775 25348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:30:21.652467 25348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:30:21.653650 25348 hybrid_clock.cc:648] HybridClock initialized: now 1786372221653616 us; error 54 us; skew 500 ppm
I20260810 14:30:21.655596 25348 webserver.cc:460] Webserver started at http://127.24.110.2:45407/ using document root <none> and password file <none>
I20260810 14:30:21.656234 25348 fs_manager.cc:359] Metadata directory not provided
I20260810 14:30:21.656329 25348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:30:21.656569 25348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:30:21.658705 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/instance:
uuid: "ece21a73bcc443c5bf2408aa5012f9ed"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.659281 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal/instance:
uuid: "ece21a73bcc443c5bf2408aa5012f9ed"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.663305 25348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:30:21.665927 25364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.667227 25348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:30:21.667366 25348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal
uuid: "ece21a73bcc443c5bf2408aa5012f9ed"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.667486 25348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:30:21.698801 25348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:30:21.699601 25348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:30:21.699815 25348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:30:21.700538 25348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:30:21.701648 25348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:30:21.701715 25348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.701784 25348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:30:21.701833 25348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.711500 25348 rpc_server.cc:307] RPC server started. Bound to: 127.24.110.2:40339
I20260810 14:30:21.711555 25523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.110.2:40339 every 8 connection(s)
I20260810 14:30:21.712489 25348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb
I20260810 14:30:21.722425 25016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTeuddy/build/debug/bin/kudu as pid 25348
I20260810 14:30:21.722549 25016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/wal/instance
I20260810 14:30:21.723526 25525 heartbeater.cc:347] Connected to a master server at 127.24.110.62:37293
I20260810 14:30:21.723788 25525 heartbeater.cc:464] Registering TS with master...
I20260810 14:30:21.724331 25525 heartbeater.cc:511] Master 127.24.110.62:37293 requested a full tablet report, sending...
I20260810 14:30:21.724704 25016 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTeuddy/build/debug/bin/kudu
/tmp/dist-test-taskTeuddy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.110.3:0
--local_ip_for_outbound_sockets=127.24.110.3
--webserver_interface=127.24.110.3
--webserver_port=0
--tserver_master_addrs=127.24.110.62:37293
--builtin_ntp_servers=127.24.110.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260810 14:30:21.725528 25075 ts_manager.cc:194] Registered new tserver with Master: ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339)
I20260810 14:30:21.726351 25075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.110.2:36445
W20260810 14:30:21.845152 25528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:30:21.845528 25528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:30:21.845610 25528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:30:21.849679 25528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:30:21.849761 25528 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260810 14:30:21.849776 25528 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 14:30:21.849853 25528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.110.3
I20260810 14:30:21.854838 25528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.110.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.110.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.110.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.24.110.62:37293
--never_fsync=true
--heap_profile_path=/tmp/kudu.25528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.110.3
--log_dir=/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:30:21.856143 25528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:30:21.857813 25528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:30:21.868391 25535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.868391 25540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:30:21.868417 25534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:30:21.868799 25528 server_base.cc:1034] running on GCE node
I20260810 14:30:21.869244 25528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:30:21.869974 25528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:30:21.871160 25528 hybrid_clock.cc:648] HybridClock initialized: now 1786372221871124 us; error 56 us; skew 500 ppm
I20260810 14:30:21.873266 25528 webserver.cc:460] Webserver started at http://127.24.110.3:37455/ using document root <none> and password file <none>
I20260810 14:30:21.873926 25528 fs_manager.cc:359] Metadata directory not provided
I20260810 14:30:21.874049 25528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:30:21.874300 25528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:30:21.876609 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/instance:
uuid: "c1b8648e400f46c0bd187b50ad14bad2"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.877244 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal/instance:
uuid: "c1b8648e400f46c0bd187b50ad14bad2"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.881183 25528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:30:21.883754 25549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.885018 25528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:30:21.885166 25528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal
uuid: "c1b8648e400f46c0bd187b50ad14bad2"
format_stamp: "Formatted at 2026-08-10 14:30:21 on dist-test-slave-bw88"
I20260810 14:30:21.885290 25528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:30:21.899109 25528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:30:21.899844 25528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:30:21.900050 25528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:30:21.900674 25528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:30:21.901731 25528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:30:21.901803 25528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.901875 25528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:30:21.901914 25528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:30:21.911732 25528 rpc_server.cc:307] RPC server started. Bound to: 127.24.110.3:45433
I20260810 14:30:21.911761 25710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.110.3:45433 every 8 connection(s)
I20260810 14:30:21.912837 25528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/info.pb
I20260810 14:30:21.913093 25016 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTeuddy/build/debug/bin/kudu as pid 25528
I20260810 14:30:21.913183 25016 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/wal/instance
I20260810 14:30:21.924964 25711 heartbeater.cc:347] Connected to a master server at 127.24.110.62:37293
I20260810 14:30:21.925256 25711 heartbeater.cc:464] Registering TS with master...
I20260810 14:30:21.925855 25711 heartbeater.cc:511] Master 127.24.110.62:37293 requested a full tablet report, sending...
I20260810 14:30:21.927088 25075 ts_manager.cc:194] Registered new tserver with Master: c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
I20260810 14:30:21.928021 25075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.110.3:49337
I20260810 14:30:21.928931 25016 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:30:21.945986 25075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58844:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:30:21.950039 25075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:30:21.973743 25254 tablet_service.cc:1467] Processing CreateTablet for tablet c232da05961d4aee9dd34fa7f348292f (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260810 14:30:21.974630 25254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c232da05961d4aee9dd34fa7f348292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.972951 25435 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4eb6d355442d8b78ff3c2ade5355 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260810 14:30:21.972976 25438 tablet_service.cc:1467] Processing CreateTablet for tablet c232da05961d4aee9dd34fa7f348292f (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260810 14:30:21.972946 25251 tablet_service.cc:1467] Processing CreateTablet for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260810 14:30:21.976128 25251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.976218 25435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4eb6d355442d8b78ff3c2ade5355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.976563 25604 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4eb6d355442d8b78ff3c2ade5355 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260810 14:30:21.977531 25437 tablet_service.cc:1467] Processing CreateTablet for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260810 14:30:21.977883 25604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4eb6d355442d8b78ff3c2ade5355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.973196 25250 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4eb6d355442d8b78ff3c2ade5355 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260810 14:30:21.978523 25250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4eb6d355442d8b78ff3c2ade5355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.984107 25438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c232da05961d4aee9dd34fa7f348292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.978340 25605 tablet_service.cc:1467] Processing CreateTablet for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260810 14:30:21.984898 25605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.984949 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9685aeaf7c9d4fedb73c60bf8bda3c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.986083 25734 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: Bootstrap starting.
I20260810 14:30:21.987592 25735 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Bootstrap starting.
I20260810 14:30:21.988157 25736 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Bootstrap starting.
I20260810 14:30:21.978770 25606 tablet_service.cc:1467] Processing CreateTablet for tablet c232da05961d4aee9dd34fa7f348292f (DEFAULT_TABLE table=all-types-table [id=2414c7bee9fb4fa5be89077a95d7e13f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260810 14:30:21.989171 25606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c232da05961d4aee9dd34fa7f348292f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:30:21.989588 25734 tablet_bootstrap.cc:654] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:21.990711 25734 log.cc:824] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: Log is configured to *not* fsync() on all Append() calls
I20260810 14:30:21.990828 25736 tablet_bootstrap.cc:654] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:21.990834 25735 tablet_bootstrap.cc:654] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:21.991694 25735 log.cc:824] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:30:21.991708 25736 log.cc:824] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:30:21.994247 25736 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: No bootstrap required, opened a new log
I20260810 14:30:21.994247 25735 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: No bootstrap required, opened a new log
I20260810 14:30:21.994478 25736 ts_tablet_manager.cc:1397] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:30:21.994478 25735 ts_tablet_manager.cc:1397] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 14:30:21.995061 25734 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: No bootstrap required, opened a new log
I20260810 14:30:21.995281 25734 ts_tablet_manager.cc:1397] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 14:30:21.998268 25734 raft_consensus.cc:348] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:21.998535 25734 raft_consensus.cc:374] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:21.998613 25734 raft_consensus.cc:729] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Initialized, Role: FOLLOWER
I20260810 14:30:21.999027 25735 raft_consensus.cc:348] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:21.999027 25736 raft_consensus.cc:348] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:21.999145 25734 consensus_queue.cc:260] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:21.999315 25736 raft_consensus.cc:374] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:21.999397 25736 raft_consensus.cc:729] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b8648e400f46c0bd187b50ad14bad2, State: Initialized, Role: FOLLOWER
I20260810 14:30:21.999502 25735 raft_consensus.cc:374] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:21.999612 25735 raft_consensus.cc:729] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995f3fa7ef4b48fcb23c34d28d47a7b9, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.000029 25736 consensus_queue.cc:260] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.000106 25735 consensus_queue.cc:260] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.000373 25734 ts_tablet_manager.cc:1428] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 14:30:22.000499 25525 heartbeater.cc:503] Master 127.24.110.62:37293 was elected leader, sending a full tablet report...
I20260810 14:30:22.000764 25734 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed: Bootstrap starting.
I20260810 14:30:22.001283 25735 ts_tablet_manager.cc:1428] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260810 14:30:22.001283 25736 ts_tablet_manager.cc:1428] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:30:22.001663 25343 heartbeater.cc:503] Master 127.24.110.62:37293 was elected leader, sending a full tablet report...
I20260810 14:30:22.001663 25711 heartbeater.cc:503] Master 127.24.110.62:37293 was elected leader, sending a full tablet report...
I20260810 14:30:22.002032 25734 tablet_bootstrap.cc:654] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.003126 25735 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9: Bootstrap starting.
I20260810 14:30:22.003710 25736 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2: Bootstrap starting.
I20260810 14:30:22.004305 25735 tablet_bootstrap.cc:654] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.004966 25736 tablet_bootstrap.cc:654] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.006251 25735 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9: No bootstrap required, opened a new log
I20260810 14:30:22.006356 25735 ts_tablet_manager.cc:1397] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:30:22.006723 25736 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2: No bootstrap required, opened a new log
I20260810 14:30:22.006809 25736 ts_tablet_manager.cc:1397] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:30:22.007094 25735 raft_consensus.cc:348] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.007221 25735 raft_consensus.cc:374] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.007258 25735 raft_consensus.cc:729] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995f3fa7ef4b48fcb23c34d28d47a7b9, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.007396 25735 consensus_queue.cc:260] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.007443 25736 raft_consensus.cc:348] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.007565 25736 raft_consensus.cc:374] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.007601 25736 raft_consensus.cc:729] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b8648e400f46c0bd187b50ad14bad2, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.007719 25735 ts_tablet_manager.cc:1428] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:30:22.007725 25736 consensus_queue.cc:260] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.007901 25735 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Bootstrap starting.
I20260810 14:30:22.008023 25736 ts_tablet_manager.cc:1428] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:30:22.008199 25736 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2: Bootstrap starting.
I20260810 14:30:22.008457 25734 tablet_bootstrap.cc:492] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed: No bootstrap required, opened a new log
I20260810 14:30:22.008550 25734 ts_tablet_manager.cc:1397] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 14:30:22.009021 25735 tablet_bootstrap.cc:654] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.009030 25734 raft_consensus.cc:348] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.009115 25734 raft_consensus.cc:374] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.009135 25734 raft_consensus.cc:729] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.009282 25734 consensus_queue.cc:260] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.009375 25736 tablet_bootstrap.cc:654] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.009588 25734 ts_tablet_manager.cc:1428] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:30:22.009739 25734 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: Bootstrap starting.
I20260810 14:30:22.011018 25734 tablet_bootstrap.cc:654] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: Neither blocks nor log segments found. Creating new log.
I20260810 14:30:22.012611 25734 tablet_bootstrap.cc:492] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: No bootstrap required, opened a new log
I20260810 14:30:22.012712 25734 ts_tablet_manager.cc:1397] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:30:22.013140 25734 raft_consensus.cc:348] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.013255 25734 raft_consensus.cc:374] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.013306 25734 raft_consensus.cc:729] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.013456 25734 consensus_queue.cc:260] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.013738 25734 ts_tablet_manager.cc:1428] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:30:22.014338 25736 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2: No bootstrap required, opened a new log
I20260810 14:30:22.014457 25736 ts_tablet_manager.cc:1397] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 14:30:22.015010 25736 raft_consensus.cc:348] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.015138 25736 raft_consensus.cc:374] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.015175 25736 raft_consensus.cc:729] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b8648e400f46c0bd187b50ad14bad2, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.015203 25735 tablet_bootstrap.cc:492] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9: No bootstrap required, opened a new log
I20260810 14:30:22.015290 25735 ts_tablet_manager.cc:1397] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 14:30:22.015355 25736 consensus_queue.cc:260] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.015525 25744 raft_consensus.cc:482] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:30:22.015697 25744 raft_consensus.cc:504] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.015759 25736 ts_tablet_manager.cc:1428] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:30:22.015797 25735 raft_consensus.cc:348] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.015925 25735 raft_consensus.cc:374] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:30:22.015987 25735 raft_consensus.cc:729] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995f3fa7ef4b48fcb23c34d28d47a7b9, State: Initialized, Role: FOLLOWER
I20260810 14:30:22.016162 25735 consensus_queue.cc:260] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.016599 25735 ts_tablet_manager.cc:1428] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:30:22.017557 25744 leader_election.cc:288] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339)
I20260810 14:30:22.022630 25461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "c1b8648e400f46c0bd187b50ad14bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" is_pre_election: true
I20260810 14:30:22.023152 25461 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b8648e400f46c0bd187b50ad14bad2 in term 0.
I20260810 14:30:22.023861 25553 leader_election.cc:302] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b8648e400f46c0bd187b50ad14bad2, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:22.024238 25744 raft_consensus.cc:2781] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:30:22.024353 25744 raft_consensus.cc:482] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:30:22.024408 25744 raft_consensus.cc:3037] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.024554 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "c1b8648e400f46c0bd187b50ad14bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" is_pre_election: true
I20260810 14:30:22.024837 25277 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b8648e400f46c0bd187b50ad14bad2 in term 0.
I20260810 14:30:22.025691 25744 raft_consensus.cc:504] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.026050 25744 leader_election.cc:288] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [CANDIDATE]: Term 1 election: Requested vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339)
I20260810 14:30:22.026619 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "c1b8648e400f46c0bd187b50ad14bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
I20260810 14:30:22.026737 25277 raft_consensus.cc:3037] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.027927 25277 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b8648e400f46c0bd187b50ad14bad2 in term 1.
I20260810 14:30:22.028255 25552 leader_election.cc:302] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9, c1b8648e400f46c0bd187b50ad14bad2; no voters:
I20260810 14:30:22.028226 25461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "c1b8648e400f46c0bd187b50ad14bad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece21a73bcc443c5bf2408aa5012f9ed"
I20260810 14:30:22.028369 25461 raft_consensus.cc:3037] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.028476 25744 raft_consensus.cc:2781] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:30:22.028638 25744 raft_consensus.cc:686] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [term 1 LEADER]: Becoming Leader. State: Replica: c1b8648e400f46c0bd187b50ad14bad2, State: Running, Role: LEADER
I20260810 14:30:22.029011 25744 consensus_queue.cc:237] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.029585 25461 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b8648e400f46c0bd187b50ad14bad2 in term 1.
I20260810 14:30:22.032531 25075 catalog_manager.cc:5495] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3). New cstate: current_term: 1 leader_uuid: "c1b8648e400f46c0bd187b50ad14bad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } health_report { overall_health: HEALTHY } } }
I20260810 14:30:22.072384 25741 raft_consensus.cc:482] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:30:22.072535 25741 raft_consensus.cc:504] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.073823 25741 leader_election.cc:288] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
I20260810 14:30:22.078176 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9685aeaf7c9d4fedb73c60bf8bda3c32" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" is_pre_election: true
I20260810 14:30:22.078369 25277 raft_consensus.cc:2454] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 0.
I20260810 14:30:22.078783 25368 leader_election.cc:302] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:22.079041 25640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9685aeaf7c9d4fedb73c60bf8bda3c32" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b8648e400f46c0bd187b50ad14bad2" is_pre_election: true
I20260810 14:30:22.079159 25741 raft_consensus.cc:2781] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:30:22.079262 25741 raft_consensus.cc:482] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:30:22.079326 25741 raft_consensus.cc:3037] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.079317 25640 raft_consensus.cc:2454] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 0.
I20260810 14:30:22.080781 25741 raft_consensus.cc:504] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.081276 25741 leader_election.cc:288] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 election: Requested vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
I20260810 14:30:22.081761 25640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9685aeaf7c9d4fedb73c60bf8bda3c32" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b8648e400f46c0bd187b50ad14bad2"
I20260810 14:30:22.081911 25640 raft_consensus.cc:3037] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.081902 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9685aeaf7c9d4fedb73c60bf8bda3c32" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
I20260810 14:30:22.082069 25277 raft_consensus.cc:3037] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.082886 25640 raft_consensus.cc:2454] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 1.
I20260810 14:30:22.082952 25277 raft_consensus.cc:2454] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 1.
I20260810 14:30:22.083230 25367 leader_election.cc:302] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b8648e400f46c0bd187b50ad14bad2, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:22.083397 25741 raft_consensus.cc:2781] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:30:22.083647 25741 raft_consensus.cc:686] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 LEADER]: Becoming Leader. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Running, Role: LEADER
I20260810 14:30:22.084005 25741 consensus_queue.cc:237] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:22.087664 25075 catalog_manager.cc:5495] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed reported cstate change: term changed from 0 to 1, leader changed from <none> to ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2). New cstate: current_term: 1 leader_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } health_report { overall_health: UNKNOWN } } }
W20260810 14:30:22.164152 25713 tablet.cc:2369] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:30:22.213734 25526 tablet.cc:2369] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:30:22.242545 25345 tablet.cc:2369] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P 995f3fa7ef4b48fcb23c34d28d47a7b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:30:22.291074 25741 raft_consensus.cc:482] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:30:22.291239 25741 raft_consensus.cc:504] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.291718 25741 leader_election.cc:288] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433), 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075)
I20260810 14:30:22.292227 25640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c232da05961d4aee9dd34fa7f348292f" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b8648e400f46c0bd187b50ad14bad2" is_pre_election: true
I20260810 14:30:22.292264 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c232da05961d4aee9dd34fa7f348292f" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" is_pre_election: true
I20260810 14:30:22.292404 25640 raft_consensus.cc:2454] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 0.
I20260810 14:30:22.292418 25277 raft_consensus.cc:2454] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 0.
I20260810 14:30:22.292752 25367 leader_election.cc:302] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1b8648e400f46c0bd187b50ad14bad2, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:22.292964 25741 raft_consensus.cc:2781] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:30:22.293035 25741 raft_consensus.cc:482] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:30:22.293072 25741 raft_consensus.cc:3037] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.294128 25741 raft_consensus.cc:504] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.294529 25741 leader_election.cc:288] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 election: Requested vote from peers c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433), 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075)
I20260810 14:30:22.294886 25640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c232da05961d4aee9dd34fa7f348292f" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b8648e400f46c0bd187b50ad14bad2"
I20260810 14:30:22.294932 25277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c232da05961d4aee9dd34fa7f348292f" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
I20260810 14:30:22.295030 25640 raft_consensus.cc:3037] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.295096 25277 raft_consensus.cc:3037] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:30:22.295943 25277 raft_consensus.cc:2454] T c232da05961d4aee9dd34fa7f348292f P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 1.
I20260810 14:30:22.295946 25640 raft_consensus.cc:2454] T c232da05961d4aee9dd34fa7f348292f P c1b8648e400f46c0bd187b50ad14bad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 1.
I20260810 14:30:22.296283 25368 leader_election.cc:302] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9, c1b8648e400f46c0bd187b50ad14bad2, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:22.296502 25741 raft_consensus.cc:2781] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:30:22.296612 25741 raft_consensus.cc:686] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [term 1 LEADER]: Becoming Leader. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Running, Role: LEADER
I20260810 14:30:22.296779 25741 consensus_queue.cc:237] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } }
I20260810 14:30:22.298620 25075 catalog_manager.cc:5495] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed reported cstate change: term changed from 0 to 1, leader changed from <none> to ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2). New cstate: current_term: 1 leader_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } health_report { overall_health: UNKNOWN } } }
I20260810 14:30:22.413120 25759 consensus_queue.cc:1035] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:22.418959 25759 consensus_queue.cc:1035] T ccfd4eb6d355442d8b78ff3c2ade5355 P c1b8648e400f46c0bd187b50ad14bad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:22.543792 25741 consensus_queue.cc:1035] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:22.549185 25763 consensus_queue.cc:1035] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:22.732667 25763 consensus_queue.cc:1035] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:22.736364 25763 consensus_queue.cc:1035] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:30:23.967494 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling FlushMRSOp(9685aeaf7c9d4fedb73c60bf8bda3c32): perf score=1.000000
I20260810 14:30:23.996642 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling FlushMRSOp(c232da05961d4aee9dd34fa7f348292f): perf score=1.000000
I20260810 14:30:24.418975 25713 maintenance_manager.cc:419] P c1b8648e400f46c0bd187b50ad14bad2: Scheduling FlushMRSOp(9685aeaf7c9d4fedb73c60bf8bda3c32): perf score=1.937278
W20260810 14:30:24.507232 25554 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-2/data/data (512380928 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 14:30:24.529413 25554 tablet_replica_mm_ops.cc:214] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P c1b8648e400f46c0bd187b50ad14bad2: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 9685aeaf7c9d4fedb73c60bf8bda3c32: unable to open output file for column bool_val BOOL NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 9685aeaf7c9d4fedb73c60bf8bda3c32's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 14:30:24.529554 25554 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 9685aeaf7c9d4fedb73c60bf8bda3c32: unable to open output file for column bool_val BOOL NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 9685aeaf7c9d4fedb73c60bf8bda3c32's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f74770a09ad google::LogMessage::Fail() at ??:0
@ 0x7f74770a388a google::LogMessage::SendToLog() at ??:0
@ 0x7f74770a04de google::LogMessage::Flush() at ??:0
@ 0x7f74770a1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f747fe19299 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f7479256674 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f747925373c _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f747925788c _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f74792f23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f74792f2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f74792f43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f74792e3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f747a9276db start_thread at ??:0
@ 0x7f7475b9e61f clone at ??:0
W20260810 14:30:25.035986 25717 meta_cache.cc:302] tablet ccfd4eb6d355442d8b78ff3c2ade5355: replica c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433) has failed: Network error: recv got EOF from 127.24.110.3:45433 (error 108)
W20260810 14:30:25.041272 25367 connection.cc:535] client connection to 127.24.110.3:45433 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:30:25.041460 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:30:25.050377 25869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:30:25.042362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.110.3:45433 (local address 127.24.110.2:44487)
0810 14:30:25.049523 (+ 7161us) negotiation.cc:107] Waiting for socket to connect
0810 14:30:25.049620 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":7070,"thread_start_us":68,"threads_started":1}
W20260810 14:30:25.050570 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20260810 14:30:25.131639 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:30:25.184338 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:25.186041 25437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 14:30:25.206745 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: FlushMRSOp(c232da05961d4aee9dd34fa7f348292f) complete. Timing: real 1.209s user 0.669s sys 0.020s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":10409,"compiler_manager_pool.run_cpu_time_us":303487,"compiler_manager_pool.run_wall_time_us":880040,"dirs.queue_time_us":1353,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":802,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3487,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":324,"threads_started":3,"wal-append.queue_time_us":223}
I20260810 14:30:25.208372 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling FlushMRSOp(9685aeaf7c9d4fedb73c60bf8bda3c32): perf score=1.937278
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 2 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable [LEADER]
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:25.230077 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:25.413664 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: FlushMRSOp(9685aeaf7c9d4fedb73c60bf8bda3c32) complete. Timing: real 1.445s user 0.558s sys 0.047s Metrics: {"bytes_written":472622,"cfile_init":1,"compiler_manager_pool.queue_time_us":21926,"compiler_manager_pool.run_cpu_time_us":303975,"compiler_manager_pool.run_wall_time_us":747662,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1151,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3231,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":221568,"thread_start_us":342,"threads_started":3,"wal-append.queue_time_us":227}
I20260810 14:30:25.414780 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling UndoDeltaBlockGCOp(9685aeaf7c9d4fedb73c60bf8bda3c32): 62920 bytes on disk
I20260810 14:30:25.415601 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: UndoDeltaBlockGCOp(9685aeaf7c9d4fedb73c60bf8bda3c32) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260810 14:30:25.416155 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling FlushMRSOp(c232da05961d4aee9dd34fa7f348292f): perf score=1.937278
I20260810 14:30:25.448917 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:25.451116 25437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 2 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable [LEADER]
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:25.478152 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:25.592711 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: FlushMRSOp(9685aeaf7c9d4fedb73c60bf8bda3c32) complete. Timing: real 0.384s user 0.378s sys 0.004s Metrics: {"bytes_written":472622,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1684,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2948,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260810 14:30:25.593413 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling UndoDeltaBlockGCOp(9685aeaf7c9d4fedb73c60bf8bda3c32): 62920 bytes on disk
I20260810 14:30:25.594108 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: UndoDeltaBlockGCOp(9685aeaf7c9d4fedb73c60bf8bda3c32) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260810 14:30:25.594522 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling UndoDeltaBlockGCOp(c232da05961d4aee9dd34fa7f348292f): 62920 bytes on disk
I20260810 14:30:25.594844 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: UndoDeltaBlockGCOp(c232da05961d4aee9dd34fa7f348292f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20260810 14:30:25.595198 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling FlushMRSOp(ccfd4eb6d355442d8b78ff3c2ade5355): perf score=1.937278
I20260810 14:30:25.780767 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:25.783680 25437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 14:30:25.791667 25947 raft_consensus.cc:482] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1b8648e400f46c0bd187b50ad14bad2)
I20260810 14:30:25.791824 25947 raft_consensus.cc:504] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:25.792281 25947 leader_election.cc:288] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
I20260810 14:30:25.792771 25275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" is_pre_election: true
I20260810 14:30:25.792918 25275 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 1.
I20260810 14:30:25.793476 25368 leader_election.cc:302] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
I20260810 14:30:25.793776 25947 raft_consensus.cc:2781] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:30:25.793833 25947 raft_consensus.cc:482] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1b8648e400f46c0bd187b50ad14bad2)
I20260810 14:30:25.793872 25947 raft_consensus.cc:3037] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:30:25.795143 25947 raft_consensus.cc:504] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:25.795480 25947 leader_election.cc:288] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 election: Requested vote from peers 995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075), c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
I20260810 14:30:25.795977 25275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9"
I20260810 14:30:25.796114 25275 raft_consensus.cc:3037] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:30:25.797271 25275 raft_consensus.cc:2454] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece21a73bcc443c5bf2408aa5012f9ed in term 2.
I20260810 14:30:25.797416 25980 raft_consensus.cc:482] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:30:25.797519 25980 raft_consensus.cc:504] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:25.797716 25368 leader_election.cc:302] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9, ece21a73bcc443c5bf2408aa5012f9ed; no voters:
W20260810 14:30:25.799283 25367 leader_election.cc:334] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111)
W20260810 14:30:25.799414 25367 leader_election.cc:334] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111)
I20260810 14:30:25.799575 25947 raft_consensus.cc:2781] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:30:25.799664 25947 raft_consensus.cc:686] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [term 2 LEADER]: Becoming Leader. State: Replica: ece21a73bcc443c5bf2408aa5012f9ed, State: Running, Role: LEADER
I20260810 14:30:25.799806 25947 consensus_queue.cc:237] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } }
I20260810 14:30:25.802127 25073 catalog_manager.cc:5495] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed reported cstate change: term changed from 1 to 2, leader changed from c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3) to ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2). New cstate: current_term: 2 leader_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" member_type: VOTER last_known_addr { host: "127.24.110.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b8648e400f46c0bd187b50ad14bad2" member_type: VOTER last_known_addr { host: "127.24.110.3" port: 45433 } health_report { overall_health: UNKNOWN } } }
I20260810 14:30:25.803496 25980 leader_election.cc:288] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339), c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433)
W20260810 14:30:25.807283 25179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111)
I20260810 14:30:25.809855 25461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4eb6d355442d8b78ff3c2ade5355" candidate_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ece21a73bcc443c5bf2408aa5012f9ed" is_pre_election: true
W20260810 14:30:25.811264 25179 leader_election.cc:334] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111)
I20260810 14:30:25.811369 25179 leader_election.cc:302] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 995f3fa7ef4b48fcb23c34d28d47a7b9; no voters: c1b8648e400f46c0bd187b50ad14bad2, ece21a73bcc443c5bf2408aa5012f9ed
I20260810 14:30:25.811786 25980 raft_consensus.cc:2726] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 14:30:25.817329 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: FlushMRSOp(c232da05961d4aee9dd34fa7f348292f) complete. Timing: real 0.401s user 0.382s sys 0.004s Metrics: {"bytes_written":436486,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1404,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3208,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":423}
I20260810 14:30:25.818048 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling UndoDeltaBlockGCOp(c232da05961d4aee9dd34fa7f348292f): 62920 bytes on disk
I20260810 14:30:25.818462 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: UndoDeltaBlockGCOp(c232da05961d4aee9dd34fa7f348292f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20260810 14:30:25.818907 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling FlushMRSOp(ccfd4eb6d355442d8b78ff3c2ade5355): perf score=1.937278
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 2 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable [LEADER]
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:25.827037 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:25.961714 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: FlushMRSOp(ccfd4eb6d355442d8b78ff3c2ade5355) complete. Timing: real 0.366s user 0.365s sys 0.000s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":102,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1505,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2199,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260810 14:30:25.962343 25345 maintenance_manager.cc:419] P 995f3fa7ef4b48fcb23c34d28d47a7b9: Scheduling UndoDeltaBlockGCOp(ccfd4eb6d355442d8b78ff3c2ade5355): 63413 bytes on disk
I20260810 14:30:25.962810 25186 maintenance_manager.cc:643] P 995f3fa7ef4b48fcb23c34d28d47a7b9: UndoDeltaBlockGCOp(ccfd4eb6d355442d8b78ff3c2ade5355) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260810 14:30:26.144430 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: FlushMRSOp(ccfd4eb6d355442d8b78ff3c2ade5355) complete. Timing: real 0.325s user 0.323s sys 0.000s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1494,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2521,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260810 14:30:26.145112 25526 maintenance_manager.cc:419] P ece21a73bcc443c5bf2408aa5012f9ed: Scheduling UndoDeltaBlockGCOp(ccfd4eb6d355442d8b78ff3c2ade5355): 63413 bytes on disk
I20260810 14:30:26.145622 25370 maintenance_manager.cc:643] P ece21a73bcc443c5bf2408aa5012f9ed: UndoDeltaBlockGCOp(ccfd4eb6d355442d8b78ff3c2ade5355) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20260810 14:30:26.222591 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:26.222652 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:26.246845 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:26.250972 25275 raft_consensus.cc:1261] T ccfd4eb6d355442d8b78ff3c2ade5355 P 995f3fa7ef4b48fcb23c34d28d47a7b9 [term 2 FOLLOWER]: Refusing update from remote peer ece21a73bcc443c5bf2408aa5012f9ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 14:30:26.251405 25947 consensus_queue.cc:1035] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "995f3fa7ef4b48fcb23c34d28d47a7b9" member_type: VOTER last_known_addr { host: "127.24.110.1" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260810 14:30:26.254673 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:30:26.807618 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:26.808171 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:26.829770 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:27.368160 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 14:30:27.643713 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:27.643713 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:27.668774 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:28.018587 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 14:30:28.723533 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:28.723518 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:28.744329 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:28.927460 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260810 14:30:29.521564 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 14:30:29.799186 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:29.799270 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:29.819371 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:30.198343 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:30:30.199210 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 14:30:30.877009 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:30.877010 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:30.896973 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:31.605603 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260810 14:30:31.960609 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:31.960636 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:31.984069 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:32.031049 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260810 14:30:32.880520 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260810 14:30:33.043787 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:33.043916 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:33.065464 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:34.127476 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:34.127528 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:34.151176 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:34.337252 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260810 14:30:34.388173 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260810 14:30:35.222494 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20260810 14:30:35.229141 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:35.229924 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:35.256378 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:35.553876 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260810 14:30:36.314917 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:36.315058 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:36.340767 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:36.787863 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260810 14:30:36.830080 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 14:30:37.413138 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:37.413597 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:37.437636 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:37.960410 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260810 14:30:38.495648 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:38.495641 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:38.517442 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:39.080776 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260810 14:30:39.467012 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260810 14:30:39.573611 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:39.573725 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:39.594101 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:40.510802 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:30:40.518306 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260810 14:30:40.653545 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:40.653656 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:40.675010 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:41.390166 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260810 14:30:41.732043 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:41.732273 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:41.752055 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:42.057745 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 14:30:42.815114 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:42.815361 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:42.836364 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:43.168135 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260810 14:30:43.771083 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260810 14:30:43.892427 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:43.892700 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:43.918629 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:44.466105 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260810 14:30:44.975298 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:44.975380 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:45.001398 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:45.414676 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260810 14:30:45.533209 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20260810 14:30:46.058701 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:46.058928 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:46.084074 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:46.352553 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260810 14:30:46.967638 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 14:30:47.142625 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:47.142625 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:47.166851 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:48.003937 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 14:30:48.230569 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:48.230623 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:48.248950 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:48.443099 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260810 14:30:49.305866 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:49.305867 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:49.332614 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:49.669448 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260810 14:30:50.388253 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:50.388579 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:50.412096 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:50.537837 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:30:50.543687 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260810 14:30:51.035723 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260810 14:30:51.468288 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:51.468549 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:51.497315 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:52.419140 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 14:30:52.552203 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:52.552259 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:52.574227 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:53.337895 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260810 14:30:53.506832 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260810 14:30:53.635928 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:53.635950 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:53.656292 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:30:54.712675 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:54.712702 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:54.735961 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:55.001602 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260810 14:30:55.712421 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:30:55.763813 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260810 14:30:55.795459 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:55.795764 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:55.817199 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:56.090596 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260810 14:30:56.877063 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:30:56.877086 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:56.902985 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:57.372239 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260810 14:30:57.958758 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:57.959613 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:57.981024 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:58.046761 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260810 14:30:58.512652 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260810 14:30:59.045881 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:30:59.045972 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:30:59.067997 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:30:59.816030 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 14:31:00.130411 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:00.130411 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:00.150329 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:00.731825 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:31:00.732637 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260810 14:31:00.887955 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260810 14:31:01.208832 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:01.209127 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:01.229485 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:02.261981 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260810 14:31:02.289557 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:02.289649 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:02.310186 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:03.014557 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260810 14:31:03.283262 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260810 14:31:03.368408 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:03.368641 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:03.400028 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:04.462774 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:04.462822 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:04.484337 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:05.047092 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 14:31:05.509008 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260810 14:31:05.557726 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:05.558252 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:05.586364 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:05.642194 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260810 14:31:05.903498 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20260810 14:31:06.643016 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:06.643069 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:06.661962 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:07.564206 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260810 14:31:07.720918 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:07.721244 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:07.743799 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:07.957844 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260810 14:31:08.160156 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260810 14:31:08.800220 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:08.800876 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:08.821086 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:09.886121 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:09.886360 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:09.907366 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:10.026022 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260810 14:31:10.614171 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260810 14:31:10.777834 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260810 14:31:10.964804 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:10.966040 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:10.990850 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:11.082309 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260810 14:31:12.049299 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:12.049299 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:12.071367 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:12.683920 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260810 14:31:13.027530 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260810 14:31:13.132083 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:13.132074 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:13.153723 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:13.307498 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260810 14:31:14.211133 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:14.211560 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:14.235517 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:15.300024 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:15.300414 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:15.322129 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:15.338585 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260810 14:31:15.373085 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260810 14:31:15.818184 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260810 14:31:16.199350 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20260810 14:31:16.380035 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:16.380147 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:16.400843 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:17.459728 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:17.459972 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:17.486683 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:17.766038 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260810 14:31:17.887241 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260810 14:31:18.348352 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260810 14:31:18.546321 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:18.546384 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:18.568382 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:19.625495 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:19.625654 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:19.646488 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:20.155088 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260810 14:31:20.342604 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260810 14:31:20.717329 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:20.717329 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:20.741962 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:20.750952 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260810 14:31:21.283843 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20260810 14:31:21.800458 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:21.800524 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:21.819531 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:22.491008 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 14:31:22.878969 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:22.879051 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:22.900702 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:23.115199 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260810 14:31:23.162286 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260810 14:31:23.958734 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:23.959151 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:23.987911 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:24.773702 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260810 14:31:25.044807 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:25.045248 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:25.069283 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:25.450793 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260810 14:31:25.812026 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260810 14:31:26.129287 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:26.129288 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:26.150020 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:26.367389 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260810 14:31:27.208845 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:27.208837 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:27.231423 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:27.341316 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260810 14:31:28.041595 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 14:31:28.288152 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:28.288729 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:28.314419 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:28.341405 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260810 14:31:29.371155 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:29.371160 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:29.392246 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:29.735323 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260810 14:31:30.485795 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:30.485832 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:30.523865 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:30.655476 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260810 14:31:31.258482 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260810 14:31:31.584892 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:31.584924 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20260810 14:31:31.590438 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:31.613837 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:32.046186 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260810 14:31:32.669065 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:32.669422 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:32.689808 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:33.024880 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260810 14:31:33.713949 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260810 14:31:33.744092 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:33.744092 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:33.765795 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:34.701563 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260810 14:31:34.822535 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:34.822664 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:34.845017 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:35.572631 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260810 14:31:35.915948 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:35.929106 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:35.956565 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:36.001767 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260810 14:31:36.594192 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260810 14:31:37.018824 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:37.019107 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:37.039597 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:37.200719 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260810 14:31:38.095528 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:38.095830 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:38.117470 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:38.179818 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260810 14:31:38.795571 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260810 14:31:39.172111 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:39.172673 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:39.193096 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:39.710996 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 14:31:40.249238 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:40.249373 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:40.269479 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:40.595386 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260810 14:31:41.324226 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:41.324226 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:41.346550 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:41.470255 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260810 14:31:41.688963 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:31:42.144212 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 14:31:42.409153 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:42.409258 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:42.434614 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:43.245440 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 14:31:43.492174 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:43.492375 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:43.513916 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:43.963621 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260810 14:31:44.568507 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:44.568507 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:44.589380 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:44.600365 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260810 14:31:45.643517 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:45.643517 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:45.663143 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:45.826153 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260810 14:31:46.591727 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260810 14:31:46.715379 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:46.715896 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:46.736500 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:47.015687 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:31:47.099047 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 14:31:47.796693 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:47.797224 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:47.824481 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:48.553977 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260810 14:31:48.880950 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:48.881139 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:48.902418 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:49.214043 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260810 14:31:49.729280 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260810 14:31:49.956640 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:49.957258 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:49.977284 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:31:51.031672 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:51.031678 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:51.051427 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:51.137252 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260810 14:31:51.891506 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260810 14:31:52.106387 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260810 14:31:52.109248 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:52.109524 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:52.130051 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:52.173439 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 14:31:53.189993 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:53.189993 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:53.212607 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:53.499857 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260810 14:31:54.267128 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:54.267297 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:54.291280 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:54.397138 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260810 14:31:54.799077 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260810 14:31:55.344465 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:55.345181 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:55.363646 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:55.748232 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260810 14:31:56.416162 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:56.416162 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:56.441661 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:56.866082 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260810 14:31:57.308130 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:31:57.329792 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 14:31:57.496698 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:31:57.496803 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:57.521081 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:58.290508 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 14:31:58.581193 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:58.581272 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:58.601652 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:59.378665 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260810 14:31:59.656697 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:31:59.656713 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:31:59.678033 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:31:59.828006 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260810 14:32:00.736779 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:00.737092 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:00.760376 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:01.060968 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260810 14:32:01.816644 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:01.816705 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:01.842665 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:01.904297 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260810 14:32:02.249470 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260810 14:32:02.383033 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20260810 14:32:02.901219 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:02.901325 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:02.922833 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:03.602862 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260810 14:32:03.985838 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:03.986660 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:04.007851 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:04.433943 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260810 14:32:04.998971 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 14:32:05.063742 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:05.063742 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:05.082626 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:05.886456 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 14:32:06.146391 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:06.146411 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:06.171300 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:06.755115 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260810 14:32:07.228430 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:07.228446 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:07.248471 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:07.527215 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:32:07.528059 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 14:32:08.302693 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:08.303570 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:08.328140 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:08.478173 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260810 14:32:09.383450 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260810 14:32:09.388350 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:09.388350 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:09.412632 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:09.967242 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 14:32:10.469660 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:10.470371 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:10.492372 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:10.958797 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260810 14:32:11.548512 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:11.548542 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:11.571596 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:11.832185 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260810 14:32:12.583564 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260810 14:32:12.636502 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:12.636910 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:12.655546 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:12.678529 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260810 14:32:13.504225 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260810 14:32:13.711318 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:13.711527 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:13.737573 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:14.158797 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260810 14:32:14.798023 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:14.798243 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:14.821413 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:15.255035 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 14:32:15.878530 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:15.878530 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:15.902053 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:15.970417 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260810 14:32:16.589308 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260810 14:32:16.957201 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:16.957571 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:16.976501 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:17.817835 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:32:17.929787 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260810 14:32:18.032577 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 14:32:18.032779 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.2:40339, tserver@127.24.110.1:43075
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:18.052145 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:18.368243 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260810 14:32:19.001295 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260810 14:32:19.116314 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:19.117070 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:19.142550 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:20.192221 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260810 14:32:20.206074 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:20.206900 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:20.232527 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260810 14:32:20.586153 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260810 14:32:21.288584 25251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 14:32:21.288910 25437 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
077af14381a74f028ac4e1ae1fd1fe01 | 127.24.110.62:37293 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.24.110.20:35861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
995f3fa7ef4b48fcb23c34d28d47a7b9 | 127.24.110.1:43075 | HEALTHY | <none> | 0 | 0
ece21a73bcc443c5bf2408aa5012f9ed | 127.24.110.2:40339 | HEALTHY | <none> | 3 | 0
c1b8648e400f46c0bd187b50ad14bad2 | 127.24.110.3:45433 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.24.110.3:45433: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.24.110.1:43075, 127.24.110.2:40339
ipki_server_key_size | 768 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
local_ip_for_outbound_sockets | 127.24.110.1 | experimental | 127.24.110.1:43075
local_ip_for_outbound_sockets | 127.24.110.2 | experimental | 127.24.110.2:40339
never_fsync | true | unsafe,advanced | 127.24.110.1:43075, 127.24.110.2:40339
openssl_security_level_override | 0 | unsafe,hidden | 127.24.110.1:43075, 127.24.110.2:40339
rpc_server_allow_ephemeral_ports | true | unsafe | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_format | pb | hidden | 127.24.110.1:43075, 127.24.110.2:40339
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.24.110.1:43075
server_dump_info_path | /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.24.110.2:40339
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.24.110.1:43075, 127.24.110.2:40339
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+----------------------------------------
builtin_ntp_servers | 127.24.110.20:35861 | 127.24.110.1:43075, 127.24.110.2:40339
time_source | builtin | 127.24.110.1:43075, 127.24.110.2:40339
Version Summary
Version | Servers
-----------------+------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.24.110.62:37293, tserver@127.24.110.1:43075, tserver@127.24.110.2:40339
Tablet Summary
Tablet c232da05961d4aee9dd34fa7f348292f of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
All reported replicas are:
A = ece21a73bcc443c5bf2408aa5012f9ed
B = c1b8648e400f46c0bd187b50ad14bad2
C = 995f3fa7ef4b48fcb23c34d28d47a7b9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9685aeaf7c9d4fedb73c60bf8bda3c32 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet ccfd4eb6d355442d8b78ff3c2ade5355 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
995f3fa7ef4b48fcb23c34d28d47a7b9 (127.24.110.1:43075): RUNNING
ece21a73bcc443c5bf2408aa5012f9ed (127.24.110.2:40339): RUNNING [LEADER]
c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): TS unavailable
All reported replicas are:
A = 995f3fa7ef4b48fcb23c34d28d47a7b9
B = ece21a73bcc443c5bf2408aa5012f9ed
C = c1b8648e400f46c0bd187b50ad14bad2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 14:32:21.309191 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 14:32:21.359588 25059 maintenance_manager.cc:419] P 077af14381a74f028ac4e1ae1fd1fe01: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371
W20260810 14:32:21.362075 25058 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskTeuddy/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786372220881473-25016-0/minicluster-data/master-0/data/data (507535360 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 14:32:21.362684 25058 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 077af14381a74f028ac4e1ae1fd1fe01: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260810 14:32:21.362759 25058 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f73f5da19ad google::LogMessage::Fail() at ??:0
@ 0x7f73f5da488a google::LogMessage::SendToLog() at ??:0
@ 0x7f73f5da14de google::LogMessage::Flush() at ??:0
@ 0x7f73f5da2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73feb1a299 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f73f7f57674 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f73f7f5473c _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f73f7f5888c _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f73f7ff33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73f7ff3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f73f7ff53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f73f7fe4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73f96286db start_thread at ??:0
@ 0x7f73f489f61f clone at ??:0
W20260810 14:32:21.454493 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260810 14:32:21.545960 25178 connection.cc:535] client connection to 127.24.110.62:37293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:32:21.545922 25365 connection.cc:535] client connection to 127.24.110.62:37293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:32:21.546398 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:32:21.546398 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:32:22.911759 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:32:22.995661 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 14:32:23.137136 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 14:32:23.783080 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260810 14:32:25.295701 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:32:25.699681 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:32:26.221041 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260810 14:32:27.798323 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:32:28.102833 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:32:28.103629 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:32:28.854446 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260810 14:32:30.279140 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:32:30.832269 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:32:31.287647 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260810 14:32:32.928289 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:32:33.107800 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:32:33.643682 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:32:33.644135 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260810 14:32:35.307781 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:32:35.954998 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:32:36.371477 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260810 14:32:37.766642 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:32:38.164630 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:32:38.165448 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:32:38.883327 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260810 14:32:40.525250 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:32:40.557389 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:32:41.451488 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260810 14:32:43.053483 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:32:43.208307 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:32:43.209235 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:32:43.945150 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260810 14:32:45.155274 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:32:45.712235 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:32:46.611579 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260810 14:32:47.604935 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:32:48.476677 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:32:48.477494 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:32:49.285889 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260810 14:32:50.299933 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:32:50.902427 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:32:51.763089 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260810 14:32:52.909102 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:32:53.467108 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:32:53.860961 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:32:54.430781 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260810 14:32:55.261029 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:32:55.860205 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:32:56.973140 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260810 14:32:57.416706 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:32:58.454200 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:32:58.965763 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:32:59.804952 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260810 14:33:00.015126 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:33:01.000810 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:33:02.196408 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260810 14:33:02.418074 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:33:03.510851 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:33:03.981889 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:33:04.786120 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260810 14:33:05.066963 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:33:05.817586 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:33:07.460554 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260810 14:33:07.524006 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:33:08.424357 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:33:08.986340 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:33:09.881162 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:33:10.018146 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260810 14:33:11.100396 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:33:12.506019 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:33:12.737046 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260810 14:33:13.623028 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:33:14.049113 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:33:14.882151 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:33:15.364082 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260810 14:33:16.127043 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:33:17.194496 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:33:17.814034 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260810 14:33:18.658377 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:33:19.197732 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:33:19.906623 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:33:20.305629 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260810 14:33:21.238972 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:33:21.621762 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (60 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:33:21.621814 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (60 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:33:22.183898 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:33:22.549460 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:33:22.788756 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260810 14:33:23.710723 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 14:33:24.363535 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:33:24.558677 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:33:25.249019 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260810 14:33:26.037629 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:33:26.970818 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:33:27.857167 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260810 14:33:28.589936 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:33:29.462792 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:33:29.484786 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:33:30.478423 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260810 14:33:31.175386 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:33:31.736559 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:33:32.872519 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260810 14:33:33.463598 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:33:34.307461 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:33:34.478581 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:33:35.697396 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260810 14:33:36.015084 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:33:36.911194 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:33:38.115584 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260810 14:33:38.620263 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:33:39.532305 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:33:39.533056 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:33:40.708979 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260810 14:33:41.206182 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:33:41.940630 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:33:43.210614 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260810 14:33:43.701618 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:33:44.635089 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:33:44.635913 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:33:45.596953 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260810 14:33:46.500792 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:33:46.985553 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:33:47.825994 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260810 14:33:49.292810 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:33:49.653234 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:33:49.654039 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:33:50.482504 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260810 14:33:52.009059 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:33:52.012635 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:33:52.820596 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260810 14:33:54.324596 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:33:54.538265 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:33:54.730515 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:33:55.132153 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260810 14:33:56.857105 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:33:57.321611 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:33:57.586620 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260810 14:33:59.576232 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:33:59.795814 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:33:59.804126 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:34:00.305824 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260810 14:34:02.041738 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:34:02.247417 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:34:02.717440 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260810 14:34:04.480531 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:34:04.515693 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:34:04.968650 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:34:05.595001 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260810 14:34:07.100440 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:34:07.102977 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:34:08.032725 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260810 14:34:09.670064 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:34:09.853089 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:34:10.070186 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:34:10.664216 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260810 14:34:12.126812 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:34:12.176867 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:34:13.449923 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260810 14:34:14.324601 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:34:14.429267 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:34:15.267745 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:34:15.844939 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260810 14:34:16.866328 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:34:17.078648 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:34:18.535748 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260810 14:34:19.420073 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 14:34:19.627022 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:34:20.536526 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:34:20.963984 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260810 14:34:21.693615 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (120 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:34:21.693743 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (120 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:34:22.038631 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260810 14:34:22.154150 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 14:34:23.471467 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:34:23.732126 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:34:24.545539 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:34:24.705231 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 14:34:25.820915 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:34:25.911105 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260810 14:34:27.076716 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:34:27.177977 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:34:28.452292 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260810 14:34:29.515928 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:34:29.743337 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:34:30.837749 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:34:31.284956 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260810 14:34:31.882570 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:34:32.309804 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:34:33.505282 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260810 14:34:34.280087 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:34:34.651683 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:34:36.094645 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260810 14:34:36.231592 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260810 14:34:36.729229 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:34:37.032527 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:34:38.508994 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260810 14:34:39.443032 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:34:39.882031 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:34:40.912495 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260810 14:34:41.187125 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:34:42.051476 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:34:42.373956 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:34:43.547425 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260810 14:34:44.589781 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:34:44.866925 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:34:45.838044 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260810 14:34:46.326833 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:34:46.781170 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:34:47.486455 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:34:48.486409 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260810 14:34:49.289932 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:34:50.261710 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:34:51.004400 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260810 14:34:51.419464 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:34:51.595324 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:34:53.084012 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:34:53.384199 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260810 14:34:54.120826 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:34:55.688266 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:34:55.825528 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260810 14:34:56.396833 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:34:56.657131 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:34:58.150683 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:34:58.260524 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260810 14:34:58.822464 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:35:00.738556 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:35:01.015360 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260810 14:35:01.580444 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:35:01.828528 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:35:03.432268 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:35:03.507936 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260810 14:35:04.277661 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:35:05.659657 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:35:05.886821 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260810 14:35:06.803848 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:35:07.030020 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:35:07.979250 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:35:08.603752 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260810 14:35:09.083525 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:35:10.455585 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:35:11.213184 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260810 14:35:11.613288 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:35:12.142834 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:35:12.956413 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:35:13.781157 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260810 14:35:13.998414 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:35:15.396998 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:35:16.066838 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260810 14:35:16.723266 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:35:17.192480 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:35:17.793143 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:35:18.621086 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260810 14:35:19.238897 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:35:20.410929 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:35:21.084523 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260810 14:35:21.638299 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 14:35:21.764207 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (180 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:35:21.764304 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (180 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:35:22.253523 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:35:22.874619 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:35:23.431950 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260810 14:35:24.256681 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20260810 14:35:24.281652 30197 consensus_queue.cc:579] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Leader has been unable to successfully communicate with peer c1b8648e400f46c0bd187b50ad14bad2 for more than 300 seconds (300.144s)
I20260810 14:35:24.795713 30197 consensus_queue.cc:579] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Leader has been unable to successfully communicate with peer c1b8648e400f46c0bd187b50ad14bad2 for more than 300 seconds (300.233s)
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:35:24.926992 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:35:25.247773 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260810 14:35:25.755403 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260810 14:35:26.166806 30251 consensus_queue.cc:579] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed [LEADER]: Leader has been unable to successfully communicate with peer c1b8648e400f46c0bd187b50ad14bad2 for more than 300 seconds (300.366s)
W20260810 14:35:26.607797 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 14:35:27.474265 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:35:27.743400 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 14:35:27.910017 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260810 14:35:28.919998 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:35:30.097386 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 14:35:30.466816 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260810 14:35:31.504899 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:35:32.408339 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:35:32.640442 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:35:32.676561 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260810 14:35:34.013258 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:35:34.630926 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:35:35.020408 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260810 14:35:36.236617 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:35:37.178057 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:35:37.646121 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:35:37.655637 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260810 14:35:38.865469 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:35:39.511189 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:35:40.100356 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260810 14:35:41.416162 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:35:42.179431 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:35:42.492775 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260810 14:35:42.984764 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:35:44.011713 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:35:44.328389 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:35:45.112830 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260810 14:35:46.257077 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:35:46.966743 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:35:47.615289 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260810 14:35:48.148160 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:35:48.820029 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:35:49.560212 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:35:50.147585 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260810 14:35:51.322718 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:35:52.155712 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:35:53.036631 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260810 14:35:53.518225 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:35:53.519174 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:35:54.619319 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:35:55.839296 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260810 14:35:56.032172 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:35:57.271301 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:35:58.166159 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260810 14:35:58.685487 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:35:58.757956 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:35:59.676959 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:36:00.695991 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260810 14:36:01.321903 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:36:02.196960 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:36:03.392441 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260810 14:36:03.758420 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:36:03.834272 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:36:04.832324 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:36:06.059190 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260810 14:36:06.647430 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:36:07.255779 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:36:08.282922 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260810 14:36:08.848038 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:36:09.061153 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:36:09.564816 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:36:10.697942 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260810 14:36:11.530514 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:36:12.115339 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:36:13.244361 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260810 14:36:13.846047 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:36:13.948068 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:36:14.535115 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:36:15.506556 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260810 14:36:16.231223 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:36:17.112947 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:36:18.021428 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260810 14:36:18.692054 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:36:19.220428 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:36:19.411890 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:36:20.384181 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260810 14:36:21.037642 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:36:21.789024 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:36:21.847510 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (240 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:36:22.235162 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (242 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:36:22.796674 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260810 14:36:23.777997 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:36:24.377601 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:36:24.408020 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:36:25.243579 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260810 14:36:26.077436 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:36:26.145810 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:36:26.825016 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:36:27.628397 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260810 14:36:28.459233 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:36:29.306476 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 14:36:29.409793 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:36:30.401371 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260810 14:36:30.793256 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:36:31.521723 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:36:32.862155 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260810 14:36:33.338819 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:36:33.969089 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:36:34.440141 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:36:35.488874 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260810 14:36:35.874394 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:36:36.299088 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:36:38.053246 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260810 14:36:38.513041 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:36:38.868021 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:36:39.500396 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:36:40.572326 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260810 14:36:41.088712 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:36:41.547385 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:36:42.900826 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260810 14:36:43.476142 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:36:44.193416 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:36:44.580704 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:36:45.326928 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260810 14:36:46.033995 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:36:46.640543 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:36:48.043782 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260810 14:36:48.691461 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:36:49.250820 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:36:49.597460 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:36:50.502064 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260810 14:36:51.127775 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:36:51.604347 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:36:53.085538 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260810 14:36:53.580502 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:36:53.892407 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:36:54.603920 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:36:55.608654 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260810 14:36:55.897020 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:36:56.628223 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:36:58.214311 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260810 14:36:58.277892 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:36:58.964815 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:36:59.811359 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:37:00.777604 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260810 14:37:00.988344 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:37:01.164070 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:37:03.371866 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260810 14:37:03.516422 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:37:03.819447 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:37:04.842231 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:37:05.698230 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260810 14:37:06.043524 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:37:06.351307 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:37:08.537154 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260810 14:37:08.702901 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:37:08.751019 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:37:09.999970 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:37:10.816936 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260810 14:37:11.147217 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:37:11.391762 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:37:13.341408 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260810 14:37:13.802933 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:37:14.072090 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:37:15.176033 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:37:16.047083 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260810 14:37:16.210039 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:37:16.417845 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:37:18.616330 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260810 14:37:18.652731 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:37:19.140980 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:37:20.381935 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:37:21.018929 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:37:21.518783 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260810 14:37:21.714538 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 14:37:21.928498 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (300 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:37:22.306949 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (302 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:37:23.876546 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260810 14:37:24.205456 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260810 14:37:24.234644 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:37:25.427947 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:37:26.247923 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:37:26.552644 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:37:26.883870 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:37:27.376729 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:37:28.814320 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:37:29.070363 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:37:29.401697 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260810 14:37:30.551273 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:37:31.302680 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:37:31.668159 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:37:31.862927 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260810 14:37:33.857674 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:37:34.281862 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:37:34.427708 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260810 14:37:35.689532 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:37:36.422962 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:37:36.578133 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:37:36.627302 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260810 14:37:38.865063 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:37:38.870903 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:37:39.479483 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260810 14:37:40.858930 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:37:41.049273 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:37:41.259167 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:37:41.937847 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260810 14:37:43.420244 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:37:43.744359 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:37:44.261575 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260810 14:37:46.020310 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:37:46.021170 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:37:46.257124 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:37:46.706099 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260810 14:37:48.630920 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:37:48.832636 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:37:49.388221 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260810 14:37:51.373710 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:37:51.587004 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:37:51.588047 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:37:51.834115 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260810 14:37:54.082247 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:37:54.218652 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:37:54.420256 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260810 14:37:56.245898 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:37:56.462738 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:37:56.534644 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:37:57.035527 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260810 14:37:58.682336 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:37:59.024390 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:37:59.461830 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260810 14:38:01.238600 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:38:01.446305 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:38:01.510017 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:02.000401 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260810 14:38:03.688858 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:38:03.776288 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:38:04.439491 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260810 14:38:06.371773 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:38:06.381881 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:38:06.779682 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:06.967270 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260810 14:38:08.684567 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:38:08.991714 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:38:09.369213 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260810 14:38:11.222963 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:38:11.374331 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:38:11.795234 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:38:11.837963 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260810 14:38:13.956766 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:38:13.963940 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:38:14.426132 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260810 14:38:16.269727 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:38:16.317019 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:38:16.814540 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:16.862708 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260810 14:38:18.315112 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:38:19.047024 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:38:19.556532 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260810 14:38:20.533458 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:38:21.575083 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:38:22.016870 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (360 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:38:22.102458 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:22.185833 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260810 14:38:22.379127 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (362 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:38:23.061571 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:38:24.055106 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:38:24.804602 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260810 14:38:25.475667 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 14:38:26.628183 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:38:27.219362 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:27.495319 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260810 14:38:27.701839 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:38:28.572770 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:38:29.356681 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 14:38:29.850553 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260810 14:38:30.370776 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:38:31.695957 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 14:38:32.227873 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:38:32.534847 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260810 14:38:32.988824 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:38:34.205437 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:38:34.893242 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260810 14:38:35.419488 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:38:36.670499 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:38:37.264179 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:38:37.728374 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260810 14:38:37.904239 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:38:39.396550 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:38:39.998641 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260810 14:38:40.309764 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:38:41.879619 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:38:42.302554 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:42.437032 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260810 14:38:42.472224 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:38:44.400555 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:38:44.755702 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260810 14:38:45.131017 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:38:46.763837 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:38:47.393167 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:38:47.394004 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260810 14:38:47.467109 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:38:49.389747 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:38:49.780292 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260810 14:38:49.841604 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:38:51.608975 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:38:52.277087 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:38:52.324445 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260810 14:38:52.572610 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:38:54.000160 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:38:54.532584 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260810 14:38:54.961740 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:38:56.416093 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:38:56.985512 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260810 14:38:57.559366 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:38:57.842945 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:38:58.948850 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:38:59.488448 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260810 14:39:00.139421 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:39:01.480934 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:39:01.827610 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260810 14:39:02.769481 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:39:02.928678 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:39:03.860751 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:39:04.102347 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260810 14:39:04.995348 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:39:06.347972 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:39:06.783330 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260810 14:39:07.499183 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:39:08.055202 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:39:09.039939 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:39:09.067873 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260810 14:39:10.070143 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:39:11.629105 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:39:11.632292 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260810 14:39:12.486133 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:39:13.058877 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:39:14.022990 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:39:14.060141 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260810 14:39:14.871061 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:39:16.664047 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260810 14:39:16.747679 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:39:17.734269 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:39:18.106236 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:39:18.931512 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:39:19.155786 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260810 14:39:20.171504 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:39:21.751624 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260810 14:39:21.797972 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:39:22.098791 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (420 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:39:22.449852 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (422 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:39:22.601558 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 14:39:23.307348 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:39:24.360014 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:39:24.439810 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260810 14:39:25.180439 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 14:39:26.771962 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260810 14:39:26.965142 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260810 14:39:27.686707 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:39:28.344424 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:39:29.340978 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 14:39:29.573040 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:39:29.773494 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:39:30.202384 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:39:31.986433 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:39:32.149398 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260810 14:39:32.747195 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:39:33.715776 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:39:34.444922 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:39:34.701774 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260810 14:39:35.363152 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:39:37.151976 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:39:37.361398 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260810 14:39:38.002696 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:39:38.732491 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:39:39.573557 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:39:39.832361 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260810 14:39:40.771572 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:39:41.972620 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:39:42.514257 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260810 14:39:43.511451 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:39:43.962293 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:39:44.508883 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:39:45.008731 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260810 14:39:45.949921 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:39:46.986553 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:39:47.743115 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260810 14:39:48.465378 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:39:49.145516 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:39:49.496655 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:39:50.231586 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260810 14:39:50.739738 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:39:51.586738 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:39:52.612579 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260810 14:39:53.264577 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:39:54.067225 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:39:54.161059 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:39:55.357776 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260810 14:39:55.708534 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:39:56.797142 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:39:58.102658 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:39:58.116768 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260810 14:39:59.221228 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:39:59.464978 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:40:00.598675 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260810 14:40:00.626461 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:40:02.315419 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:40:03.300767 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:40:03.459794 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260810 14:40:04.283625 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20260810 14:40:04.899134 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:40:05.736533 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:40:05.904850 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260810 14:40:07.838806 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:40:08.374963 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:40:08.442533 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260810 14:40:09.337517 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:40:10.462309 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:40:10.968632 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:40:11.084323 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260810 14:40:13.178849 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:40:13.389613 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:40:13.708603 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260810 14:40:14.346637 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:15.599097 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:40:16.005868 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:40:16.022207 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260810 14:40:18.057010 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:40:18.555703 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:40:18.606475 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260810 14:40:19.443421 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:20.481890 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:40:21.109104 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260810 14:40:21.243221 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:40:22.180820 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (480 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:40:22.521159 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (482 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:40:22.845182 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:40:23.609838 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 14:40:23.888032 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260810 14:40:24.669239 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:40:25.137391 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:40:26.161507 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 14:40:26.171959 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260810 14:40:27.498513 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 14:40:28.476941 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260810 14:40:28.680841 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:40:29.736802 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:40:30.008932 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:40:30.947208 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:40:30.993723 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260810 14:40:31.392473 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:40:32.449331 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:40:33.637712 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260810 14:40:33.928035 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:40:34.871997 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:34.872984 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:40:36.026580 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260810 14:40:36.286209 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:40:37.268502 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:40:38.605742 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:40:38.639603 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260810 14:40:39.771452 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:40:39.991720 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:40:40.886063 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260810 14:40:41.005985 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:40:42.487898 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:40:43.504083 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:40:43.546005 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260810 14:40:44.995465 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:45.240906 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:40:46.044909 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:40:46.046425 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260810 14:40:47.538589 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:40:48.796830 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260810 14:40:48.853978 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:40:49.880429 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:40:50.136206 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:51.281773 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:40:51.437319 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260810 14:40:52.518412 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:40:53.743214 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260810 14:40:53.891906 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:40:54.934099 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:40:55.295639 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:40:56.216014 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260810 14:40:56.407763 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:40:57.392318 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:40:58.991101 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260810 14:40:59.129562 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:40:59.789310 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:41:00.317873 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:41:01.409098 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260810 14:41:01.701114 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:41:02.458261 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:41:03.825524 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260810 14:41:04.035179 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:41:04.949440 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:41:05.485508 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:41:06.494514 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260810 14:41:06.528649 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:41:07.580896 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:41:08.856452 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:41:09.079104 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260810 14:41:10.195222 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:41:10.565534 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:41:11.517903 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:41:11.628566 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260810 14:41:12.960855 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:41:13.953485 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:41:14.092237 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260810 14:41:15.570647 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:41:15.571555 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:41:16.544467 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260810 14:41:16.659044 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:41:18.085989 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:41:18.888932 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260810 14:41:19.038369 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:41:20.573993 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:41:20.574852 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:41:21.412338 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260810 14:41:21.633445 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:41:22.263886 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (540 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:41:22.591365 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (542 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:41:23.099181 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:41:23.734328 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260810 14:41:24.006472 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:41:25.499126 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:41:25.858686 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:41:26.262403 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260810 14:41:26.611552 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:41:28.094799 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:41:28.579288 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260810 14:41:29.153898 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 14:41:30.532025 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:41:30.996901 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:41:31.060134 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260810 14:41:31.482308 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:41:32.138716 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:41:32.761705 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 14:41:33.444620 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260810 14:41:33.843407 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:41:35.478940 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 14:41:35.818447 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260810 14:41:36.055855 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:41:36.421682 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:41:38.141238 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:41:38.324249 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260810 14:41:38.660162 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:41:40.452924 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:41:40.506222 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260810 14:41:40.910125 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:41:41.067754 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:41:42.853993 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:41:42.884898 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260810 14:41:43.451315 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:41:45.161000 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:41:45.436919 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260810 14:41:45.684441 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:41:46.250624 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:41:47.742911 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260810 14:41:47.903184 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:41:48.304859 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:41:50.398816 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:41:50.482995 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260810 14:41:50.726028 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:41:51.404588 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:41:53.134582 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:41:53.167109 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260810 14:41:53.207697 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:41:55.570209 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:41:55.683055 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:41:55.742365 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260810 14:41:56.580355 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:41:57.876912 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:41:58.056226 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260810 14:41:58.190050 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:42:00.445850 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:42:00.529258 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:42:00.654572 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260810 14:42:01.617437 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:42:02.807299 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:42:02.998956 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260810 14:42:03.131290 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:42:05.468222 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:42:05.684381 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260810 14:42:05.756649 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:42:06.773439 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:42:07.940311 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:42:08.307008 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260810 14:42:08.416785 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:42:10.541455 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:42:10.679692 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:42:10.920174 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260810 14:42:11.833524 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:42:12.987234 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:42:13.249150 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:42:13.367305 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260810 14:42:15.451265 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:42:15.794068 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260810 14:42:15.835606 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:42:17.031579 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:42:18.044687 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:42:18.084306 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:42:18.257660 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260810 14:42:20.473692 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:42:20.550446 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:42:20.684407 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260810 14:42:22.224736 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:42:22.344285 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (600 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:42:22.665148 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (602 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:42:22.923928 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:42:23.255119 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:42:23.313661 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260810 14:42:25.552274 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:42:25.821165 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:42:25.946256 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260810 14:42:27.293298 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:42:28.041288 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:42:28.065639 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:42:28.303025 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260810 14:42:30.469383 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:42:30.694612 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:42:30.832041 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260810 14:42:32.422315 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:42:32.882465 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:42:33.220544 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:42:33.326865 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:42:33.432996 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260810 14:42:35.233556 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:42:35.701821 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 14:42:35.914549 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260810 14:42:37.534736 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:42:37.935052 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:42:37.976622 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 14:42:38.682041 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260810 14:42:40.276651 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 14:42:40.414800 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:42:41.114331 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260810 14:42:42.596469 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:42:42.831409 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:42:43.107586 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:42:43.676095 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260810 14:42:45.560081 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:42:45.635308 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:42:46.426759 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260810 14:42:47.705399 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:42:48.004824 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:42:48.229068 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:42:48.713896 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260810 14:42:50.551188 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:42:50.644234 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:42:51.249894 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260810 14:42:52.770045 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:42:52.968652 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:42:53.350739 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:42:53.599839 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260810 14:42:55.079799 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:42:55.989113 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260810 14:42:56.102710 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:42:57.491896 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:42:57.924377 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:42:58.513895 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:42:58.652223 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260810 14:42:59.865082 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:43:01.075182 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:43:01.378508 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260810 14:43:02.344491 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:43:03.175109 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:43:03.652626 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:43:04.001464 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260810 14:43:05.206923 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:43:06.189666 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:43:06.822762 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260810 14:43:07.578927 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:43:08.252803 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:43:08.765856 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:43:09.266860 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260810 14:43:09.849042 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:43:11.188519 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:43:11.938215 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260810 14:43:12.399955 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:43:13.539357 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:43:13.637806 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:43:14.623051 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260810 14:43:14.737494 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:43:16.123510 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:43:17.408769 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260810 14:43:17.540534 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:43:18.420022 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:43:18.599666 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:43:20.020521 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:43:20.099133 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260810 14:43:20.690610 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:43:22.425987 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (660 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:43:22.518558 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:43:22.559510 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260810 14:43:22.736915 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (662 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:43:23.414996 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:43:23.661978 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:43:24.828554 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:43:25.266991 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260810 14:43:25.987561 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:43:27.585212 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:43:27.984135 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260810 14:43:28.402931 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:43:28.744443 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:43:30.220332 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:43:30.611307 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260810 14:43:30.802922 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 14:43:32.772954 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:43:32.849011 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260810 14:43:33.226846 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:43:33.762207 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:43:34.556181 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:43:35.095166 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260810 14:43:35.478223 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 14:43:35.547263 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:43:37.469165 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260810 14:43:38.034734 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:43:38.237996 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:43:38.855258 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:43:39.823175 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260810 14:43:40.161468 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:43:40.509936 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:43:42.490278 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260810 14:43:42.816251 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:43:43.292686 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:43:44.321846 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:43:44.736045 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260810 14:43:45.273272 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:43:45.684917 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:43:47.236095 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260810 14:43:47.849462 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:43:48.360867 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:43:49.366875 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:43:49.556939 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260810 14:43:50.625857 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:43:50.957183 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:43:52.009553 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260810 14:43:53.391625 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:43:53.693516 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:43:54.464125 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:43:54.596462 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260810 14:43:55.761627 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:43:56.123039 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:43:56.936928 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260810 14:43:57.822269 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:43:58.840155 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:43:59.499284 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:43:59.499964 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260810 14:44:00.057684 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:44:01.291168 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:44:01.773885 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260810 14:44:02.554678 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:44:03.831079 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:44:04.341527 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260810 14:44:04.694411 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:44:04.962990 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:44:06.091223 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:44:06.733286 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260810 14:44:07.366386 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:44:08.304137 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:44:09.545714 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260810 14:44:10.010885 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:44:10.011763 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:44:11.133091 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:44:12.246101 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:44:12.265787 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260810 14:44:13.467638 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:44:14.559870 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260810 14:44:14.798835 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:44:15.036816 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:44:15.967149 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:44:17.084512 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260810 14:44:17.499346 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:44:18.741679 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:44:19.524470 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260810 14:44:19.654284 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:44:20.067269 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260810 14:44:21.370949 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:44:21.890586 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260810 14:44:22.176810 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:44:22.508225 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (720 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:44:22.808097 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (722 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:44:23.707350 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:44:24.421627 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260810 14:44:24.430853 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:44:25.262642 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:44:26.250208 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:44:26.992830 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:44:27.065625 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260810 14:44:28.607244 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 14:44:29.318393 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:44:29.767258 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260810 14:44:30.344264 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:44:31.166671 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:44:31.764695 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 14:44:32.288673 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260810 14:44:33.373594 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:44:34.501466 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:44:34.965193 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260810 14:44:35.404635 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:44:35.550040 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
---------------------------------+---------------------+-------------
<unknown> (127.24.110.62:37293) | 127.24.110.62:37293 | UNAVAILABLE
Error from 127.24.110.62:37293: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
FAILED
I20260810 14:44:35.737129 25016 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.24.110.62:37293, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:44:36.699275 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:44:37.274961 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260810 14:44:38.054396 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:44:39.036232 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 14:44:39.919239 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260810 14:44:40.412916 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:44:40.557731 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:44:41.510115 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:44:42.636303 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260810 14:44:42.929905 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:44:44.246146 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:44:45.016297 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260810 14:44:45.489486 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:44:45.833230 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:44:46.569219 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:44:47.322558 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260810 14:44:48.157583 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 14:44:49.138818 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:44:49.695639 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260810 14:44:50.484957 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:44:50.713472 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
************************ BEGIN STACKS **************************
[New LWP 25019]
[New LWP 25020]
[New LWP 25024]
[New LWP 25716]
[New LWP 25717]
[New LWP 25718]
[New LWP 25720]
[New LWP 31238]
[New LWP 31239]
[New LWP 31240]
[New LWP 31241]
[New LWP 31244]
[New LWP 31245]
[New LWP 31246]
[New LWP 31248]
[New LWP 31251]
W20260810 14:44:51.884141 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 14:44:52.447266 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260810 14:44:53.029546 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:44:54.723460 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:44:55.006887 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260810 14:44:55.396797 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:44:55.769321 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:44:57.172547 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:44:57.265628 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260810 14:44:57.839816 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:44:59.474218 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:44:59.893138 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260810 14:45:00.285025 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:45:00.777076 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:45:01.917629 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:45:02.604949 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260810 14:45:02.981905 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:45:04.331059 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:45:05.188723 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260810 14:45:05.445456 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:45:05.998340 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260810 14:45:06.742615 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:45:07.735515 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260810 14:45:07.765802 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:45:08.942022 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:45:10.314661 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260810 14:45:10.340330 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 14:45:11.289291 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260810 14:45:11.293748 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:45:12.896186 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 14:45:12.931866 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260810 14:45:13.924501 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260810 14:45:15.347323 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260810 14:45:15.365391 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260810 14:45:16.369226 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:45:16.370103 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 14:45:17.703408 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260810 14:45:17.819097 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260810 14:45:19.089854 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
0x00007f8a0cb1fad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f8a0716d240 (LWP 25016) "all_types-itest" 0x00007f8a0cb1fad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f8a0690f700 (LWP 25019) "all_types-itest" 0x00007f8a0cb2432a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f8a05346700 (LWP 25020) "kernel-watcher-" 0x00007f8a0cb1ffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f8a04b45700 (LWP 25024) "rpc reactor-250" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f8a04344700 (LWP 25716) "rpc reactor-257" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f8a0393c700 (LWP 25717) "rpc reactor-257" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f8a0313b700 (LWP 25718) "rpc reactor-257" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f8a0293a700 (LWP 25720) "rpc reactor-257" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f89fb92c700 (LWP 31238) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f89fd12f700 (LWP 31239) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f8a00135700 (LWP 31240) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f89fa92a700 (LWP 31241) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f8a01137700 (LWP 31244) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f8a00936700 (LWP 31245) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f89ff934700 (LWP 31246) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f89ff133700 (LWP 31248) "rpc reactor-312" 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f89fe932700 (LWP 31251) "client-negotiat" 0x00007f8a0cb1ffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f89fe932700 (LWP 31251)):
#0 0x00007f8a0cb1ffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260810 14:45:20.090703 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260810 14:45:20.149833 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#1 0x00007f8a0b8c4de9 in kudu::ConditionVariable::WaitFor (this=0x7f89fe931140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007f8a0b9f9ecf in kudu::ThreadPool::DispatchThread (this=0x35bc1400) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f8a0b9fac83 in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x35b10028) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
#4 0x00007f8a0b9fc3f2 in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#5 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b10028) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#6 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b10000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f89ff133700 (LWP 31248)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b51b00, timeout=0.098894765600562096) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b51b00, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d653c8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d653c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b11e38) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b11e38) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b11e10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f89ff934700 (LWP 31246)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35ba9200, timeout=0.09969339775852859) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35ba9200, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d65228, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d65220) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b106a8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b106a8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b10680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f8a00936700 (LWP 31245)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35baa880, timeout=0.099682880798354745) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35baa880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d65088, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d65080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b11af8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b11af8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b11ad0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f8a01137700 (LWP 31244)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35bab600, timeout=0.087701428914442658) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35bab600, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d64ee8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d64ee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x362c09e8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x362c09e8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x362c09c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f89fa92a700 (LWP 31241)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35baba80, timeout=0.098972832784056664) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35baba80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d64ba8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d64ba0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x362c1618) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x362c1618) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x362c15f0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f8a00135700 (LWP 31240)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35ba9b00, timeout=0.098961266921833158) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35ba9b00, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d64a08, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d64a00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x362c1d68) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x362c1d68) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x362c1d40) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f89fd12f700 (LWP 31239)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35bab180, timeout=0.09928623866289854) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35bab180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d64868, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d64860) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x362c1f08) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x362c1f08) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x362c1ee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f89fb92c700 (LWP 31238)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35ba8d80, timeout=0.098915552720427513) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35ba8d80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35d646c8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35d646c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x362c0c58) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x362c0c58) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x362c0c30) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f8a0293a700 (LWP 25720)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b51680, timeout=0.098946590907871723) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b51680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35adf088, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35adf080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b116e8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b116e8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b116c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f8a0313b700 (LWP 25718)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b51200, timeout=0.098953701788559556) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b51200, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35adeee8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35adeee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b11618) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b11618) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b115f0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f8a0393c700 (LWP 25717)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b50d80, timeout=0.098932384746149182) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b50d80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35adea08, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35adea00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b11548) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b11548) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b11520) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f8a04344700 (LWP 25716)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b50900, timeout=0.098878968972712755) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b50900, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35aded48, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35aded40) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b11478) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b11478) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b11450) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f8a04b45700 (LWP 25024)):
#0 0x00007f8a07f3e947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8a0a979bda in epoll_poll (loop=0x35b50480, timeout=0.099399092840030789) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f8a0a97d228 in ev_run (loop=0x35b50480, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f8a0fc37207 in ev::loop_ref::run (this=0x35ade388, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f8a0fc3234a in kudu::rpc::ReactorThread::RunThread (this=0x35ade380) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f8a0fc2efb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x35b10368) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f8a0fc3644d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35b10368) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35b10340) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f8a05346700 (LWP 25020)):
#0 0x00007f8a0cb1ffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260810 14:45:21.557664 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260810 14:45:21.713294 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#1 0x00007f8a0b8c4c38 in kudu::ConditionVariable::WaitUntil (this=0x359c3878, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
#2 0x00007f8a1447c4e1 in kudu::CountDownLatch::WaitUntil (this=0x359c3840, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f8a1447c569 in kudu::CountDownLatch::WaitFor (this=0x359c3840, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f8a0b950f57 in kudu::KernelStackWatchdog::RunThread (this=0x359c37a0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f8a0b950291 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x35a12ec8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f8a0b9517f3 in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f8a14833170 in std::function<void ()>::operator()() const (this=0x35a12ec8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f8a0b9eb512 in kudu::Thread::SuperviseThread (arg=0x35a12ea0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f8a0690f700 (LWP 25019)):
#0 0x00007f8a0cb2432a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f8a0b9e2e73 in kudu::Subprocess::DoWait (this=0x7f8a0690e2a0, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#2 0x00007f8a0b9e0433 in kudu::Subprocess::Wait (this=0x7f8a0690e2a0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#3 0x00007f8a0b93f869 in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#4 0x00007f8a0b93f418 in kudu::PstackWatcher::RunGdbStackDump (pid=25016, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#5 0x00007f8a0b93ed1a in kudu::PstackWatcher::DumpPidStacks (pid=25016, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#6 0x00007f8a0b93ecb5 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#7 0x0000000000513498 in kudu::<lambda()>::operator()(void) const (__closure=0x359ebbc8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#8 0x0000000000513bae in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:60
#9 0x0000000000513960 in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
#10 0x0000000000514180 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x359ebbc8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
#11 0x000000000051412a in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x359ebbc8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
#12 0x00000000005140f4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x359ebbc0) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
#13 0x00007f8a10832a4f in execute_native_thread_routine () from /tmp/dist-test-taskTeuddy/build/debug/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#14 0x00007f8a0cb196db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f8a07f3e61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f8a0716d240 (LWP 25016)):
#0 0x00007f8a0cb1fad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f8a0b8c4ade in kudu::ConditionVariable::Wait (this=0x35d68cd0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
W20260810 14:45:22.591773 25343 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (780 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
W20260810 14:45:22.655484 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260810 14:45:22.741102 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#2 0x00007f8a14c4a114 in kudu::CountDownLatch::Wait (this=0x35d68c98) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260810 14:45:22.879980 25525 heartbeater.cc:644] Failed to heartbeat to 127.24.110.62:37293 (782 consecutive failures): Network error: Failed to ping master at 127.24.110.62:37293: Client connection negotiation failed: client connection to 127.24.110.62:37293: connect: Connection refused (error 111)
#3 0x00007f8a1476852a in kudu::Synchronizer::Wait (this=0x7ffc4c4c8da0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f8a12835d7d in kudu::client::KuduClient::Data::ConnectToCluster (this=0x36404000, client=0x36e3a8a0, deadline=..., creds_policy=kudu::rpc::CredentialsPolicy::ANY_CREDENTIALS) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260810 14:45:24.244834 25367 consensus_peers.cc:487] T 9685aeaf7c9d4fedb73c60bf8bda3c32 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#5 0x00007f8a127fb0e4 in kudu::client::KuduClientBuilder::Build (this=0x7ffc4c4c9158, client=0x36451310) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
W20260810 14:45:24.969465 25367 consensus_peers.cc:487] T c232da05961d4aee9dd34fa7f348292f P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260810 14:45:25.077425 25367 consensus_peers.cc:487] T ccfd4eb6d355442d8b78ff3c2ade5355 P ece21a73bcc443c5bf2408aa5012f9ed -> Peer c1b8648e400f46c0bd187b50ad14bad2 (127.24.110.3:45433): Couldn't send request to peer c1b8648e400f46c0bd187b50ad14bad2. Status: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111). This is attempt 1806: this message will repeat every 5th retry.
#6 0x00007f8a12b1abd1 in kudu::tools::CreateKuduClient (master_addresses=std::vector of length 1, capacity 1 = {...}, client=0x36451310, can_see_all_replicas=true) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
#7 0x00007f8a14492c9d in kudu::tools::RemoteKsckCluster::Connect (this=0x36451200) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:530
#8 0x00007f8a1445033f in kudu::tools::Ksck::CheckClusterRunning (this=0x35a34410) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f8a1445180a in kudu::tools::Ksck::Run (this=0x35a34410) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f8a14453a86 in kudu::tools::Ksck::RunAndPrintResults (this=0x35a34410) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
#11 0x00007f8a14c1c640 in kudu::ClusterVerifier::RunKsck (this=0x7ffc4c4c9930) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f8a14c1c0a5 in kudu::ClusterVerifier::CheckCluster (this=0x7ffc4c4c9930) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x00000000004c6511 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x35ae0900, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x000000000049ea4c in kudu::client::AllTypesItest_TestAllKeyTypes_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::TestBody (this=0x35ae0900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:676
#15 0x00007f8a0bc7cbfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f8a0bc82a28 "the test body", method=<optimized out>, object=0x35ae0900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x35ae0900, method=<optimized out>, location=location@entry=0x7f8a0bc82a28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f8a0bc715db in testing::Test::Run (this=0x35ae0900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x35ae0900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f8a0bc71795 in testing::TestInfo::Run (this=0x35a81ad0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f8a0bc71cdf in testing::TestSuite::Run (this=0x35ab5ef0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x35ab5ef0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f8a0bc7229f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f8a0bc7d13e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f8a0bc845a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x359fe280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x359fe280, method=<optimized out>, location=location@entry=0x7f8a0bc845a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f8a0bc7183d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x00000000005144b0 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x00000000005138da in main (argc=1, argv=0x7ffc4c4c9e98) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20260810 14:45:26.554085 25019 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786373126 (unix time) try "date -d @1786373126" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000061b8) received by PID 25016 (TID 0x7f8a0690f700) from PID 25016; stack trace: ***
@ 0x7f8a0a3a6a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f8a0cb24980 (unknown) at ??:0
@ 0x7f8a07e5be87 gsignal at ??:0
@ 0x7f8a07e5d7f1 abort at ??:0
@ 0x7f8a0a3989ad google::LogMessage::Fail() at ??:0
@ 0x7f8a0a39b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8a0a3984de google::LogMessage::Flush() at ??:0
@ 0x7f8a0a399a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5135cc _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63 (discriminator 9)
@ 0x513bae _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:60
@ 0x513960 _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:96
@ 0x514180 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
@ 0x51412a _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
@ 0x5140f4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
@ 0x7f8a10832a4f execute_native_thread_routine at ??:0
@ 0x7f8a0cb196db start_thread at ??:0
@ 0x7f8a07f3e61f clone at ??:0
W20260810 14:45:26.623833 25367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.110.3:45433: connect: Connection refused (error 111) [suppressed 29 similar messages]