Note: This is test shard 5 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/2, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> >
[ RUN ] AllTypesItest/2.TestAllKeyTypes
2026-08-11T01:57:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-11T01:57:22Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260811 01:57:22.108521 10588 external_mini_cluster.cc:1372] Running /tmp/dist-test-task67xTv3/build/debug/bin/kudu
/tmp/dist-test-task67xTv3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.87.62:36291
--webserver_interface=127.10.87.62
--webserver_port=0
--builtin_ntp_servers=127.10.87.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.87.62:36291 with env {}
W20260811 01:57:22.236162 10597 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 01:57:22.236428 10597 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 01:57:22.236471 10597 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260811 01:57:22.240377 10597 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 01:57:22.240453 10597 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260811 01:57:22.240473 10597 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 01:57:22.240491 10597 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260811 01:57:22.244772 10597 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.87.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.87.62:36291
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.87.62:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.87.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 11 Aug 2026 01:43:15 UTC on bdcb31816ec0
build id 13280
I20260811 01:57:22.246380 10597 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 01:57:22.248260 10597 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260811 01:57:22.259260 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.259446 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.259672 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.260578 10597 server_base.cc:1061] running on GCE node
I20260811 01:57:22.261409 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 01:57:22.263110 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 01:57:22.264322 10597 hybrid_clock.cc:648] HybridClock initialized: now 1786413442264278 us; error 59 us; skew 500 ppm
I20260811 01:57:22.266887 10597 webserver.cc:533] Webserver started at http://127.10.87.62:40843/ using document root <none> and password file <none>
I20260811 01:57:22.267829 10597 fs_manager.cc:362] Metadata directory not provided
I20260811 01:57:22.267923 10597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 01:57:22.268304 10597 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 01:57:22.270426 10597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/instance:
uuid: "b6613112d9004b319b2c743bfe4067e9"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.270931 10597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal/instance:
uuid: "b6613112d9004b319b2c743bfe4067e9"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.275959 10597 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 01:57:22.279438 10611 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.280853 10597 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 01:57:22.281029 10597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data,/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal
uuid: "b6613112d9004b319b2c743bfe4067e9"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.281148 10597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 01:57:22.297806 10597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260811 01:57:22.298628 10597 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 01:57:22.298888 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 01:57:22.309967 10666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.87.62:36291 every 8 connection(s)
I20260811 01:57:22.313020 10597 rpc_server.cc:307] RPC server started. Bound to: 127.10.87.62:36291
I20260811 01:57:22.314718 10597 server_base.cc:1193] Dumped server information to /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb
I20260811 01:57:22.316287 10588 external_mini_cluster.cc:1434] Started /tmp/dist-test-task67xTv3/build/debug/bin/kudu as pid 10597
I20260811 01:57:22.316546 10588 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/wal/instance
I20260811 01:57:22.322495 10667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:22.332403 10667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: Bootstrap starting.
I20260811 01:57:22.336580 10667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:22.337687 10667 log.cc:826] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: Log is configured to *not* fsync() on all Append() calls
I20260811 01:57:22.340237 10667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: No bootstrap required, opened a new log
I20260811 01:57:22.344012 10667 raft_consensus.cc:359] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } }
I20260811 01:57:22.344322 10667 raft_consensus.cc:385] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:22.344503 10667 raft_consensus.cc:740] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6613112d9004b319b2c743bfe4067e9, State: Initialized, Role: FOLLOWER
I20260811 01:57:22.345232 10667 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } }
I20260811 01:57:22.345474 10667 raft_consensus.cc:399] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 01:57:22.345587 10667 raft_consensus.cc:493] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 01:57:22.345777 10667 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:22.346659 10667 raft_consensus.cc:515] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } }
I20260811 01:57:22.347235 10667 leader_election.cc:304] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6613112d9004b319b2c743bfe4067e9; no voters:
I20260811 01:57:22.347651 10667 leader_election.cc:290] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 01:57:22.347851 10674 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 01:57:22.348182 10674 raft_consensus.cc:697] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [term 1 LEADER]: Becoming Leader. State: Replica: b6613112d9004b319b2c743bfe4067e9, State: Running, Role: LEADER
I20260811 01:57:22.348620 10674 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } }
I20260811 01:57:22.349206 10667 sys_catalog.cc:565] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 01:57:22.351102 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6613112d9004b319b2c743bfe4067e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } } }
I20260811 01:57:22.351320 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6613112d9004b319b2c743bfe4067e9. Latest consensus state: current_term: 1 leader_uuid: "b6613112d9004b319b2c743bfe4067e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6613112d9004b319b2c743bfe4067e9" member_type: VOTER last_known_addr { host: "127.10.87.62" port: 36291 } } }
I20260811 01:57:22.351356 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [sys.catalog]: This master's current role is: LEADER
I20260811 01:57:22.351414 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9 [sys.catalog]: This master's current role is: LEADER
I20260811 01:57:22.352649 10683 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260811 01:57:22.356707 10683 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260811 01:57:22.364094 10683 catalog_manager.cc:1383] Generated new cluster ID: d5ef313eb2604ffe8bfb76818454e13b
I20260811 01:57:22.364194 10683 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260811 01:57:22.374326 10683 catalog_manager.cc:1406] Generated new certificate authority record
I20260811 01:57:22.375233 10683 catalog_manager.cc:1540] Loading token signing keys...
I20260811 01:57:22.382827 10683 catalog_manager.cc:6092] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: Generated new TSK 0
I20260811 01:57:22.383736 10683 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260811 01:57:22.398350 10588 external_mini_cluster.cc:1372] Running /tmp/dist-test-task67xTv3/build/debug/bin/kudu
/tmp/dist-test-task67xTv3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.87.1:0
--local_ip_for_outbound_sockets=127.10.87.1
--webserver_interface=127.10.87.1
--webserver_port=0
--tserver_master_addrs=127.10.87.62:36291
--builtin_ntp_servers=127.10.87.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260811 01:57:22.511041 10694 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 01:57:22.511318 10694 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 01:57:22.511399 10694 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260811 01:57:22.514989 10694 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260811 01:57:22.515112 10694 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 01:57:22.515172 10694 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260811 01:57:22.515252 10694 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.87.1
I20260811 01:57:22.519421 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.87.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.87.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.87.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.10.87.62:36291
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.87.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 11 Aug 2026 01:43:15 UTC on bdcb31816ec0
build id 13280
I20260811 01:57:22.520646 10694 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 01:57:22.521943 10694 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260811 01:57:22.529385 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.529361 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.529597 10694 server_base.cc:1061] running on GCE node
W20260811 01:57:22.529776 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.530232 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 01:57:22.530841 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 01:57:22.532022 10694 hybrid_clock.cc:648] HybridClock initialized: now 1786413442532002 us; error 36 us; skew 500 ppm
I20260811 01:57:22.533967 10694 webserver.cc:533] Webserver started at http://127.10.87.1:42487/ using document root <none> and password file <none>
I20260811 01:57:22.534591 10694 fs_manager.cc:362] Metadata directory not provided
I20260811 01:57:22.534648 10694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 01:57:22.534889 10694 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 01:57:22.536516 10694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/instance:
uuid: "dd70368ef7ad4752afbe694f0aef10b7"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.537006 10694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal/instance:
uuid: "dd70368ef7ad4752afbe694f0aef10b7"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.540484 10694 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 01:57:22.542825 10708 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.543903 10694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 01:57:22.544020 10694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data,/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal
uuid: "dd70368ef7ad4752afbe694f0aef10b7"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.544126 10694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 01:57:22.558250 10694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260811 01:57:22.559067 10694 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 01:57:22.559275 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 01:57:22.560153 10694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260811 01:57:22.562000 10694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 01:57:22.562099 10694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.562172 10694 ts_tablet_manager.cc:616] Registered 0 tablets
I20260811 01:57:22.562263 10694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 01:57:22.573139 10694 rpc_server.cc:307] RPC server started. Bound to: 127.10.87.1:37603
I20260811 01:57:22.573160 10832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.87.1:37603 every 8 connection(s)
I20260811 01:57:22.574200 10694 server_base.cc:1193] Dumped server information to /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb
I20260811 01:57:22.575273 10588 external_mini_cluster.cc:1434] Started /tmp/dist-test-task67xTv3/build/debug/bin/kudu as pid 10694
I20260811 01:57:22.575397 10588 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/wal/instance
I20260811 01:57:22.578112 10588 external_mini_cluster.cc:1372] Running /tmp/dist-test-task67xTv3/build/debug/bin/kudu
/tmp/dist-test-task67xTv3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.87.2:0
--local_ip_for_outbound_sockets=127.10.87.2
--webserver_interface=127.10.87.2
--webserver_port=0
--tserver_master_addrs=127.10.87.62:36291
--builtin_ntp_servers=127.10.87.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 01:57:22.585671 10833 heartbeater.cc:344] Connected to a master server at 127.10.87.62:36291
I20260811 01:57:22.586005 10833 heartbeater.cc:461] Registering TS with master...
I20260811 01:57:22.586687 10833 heartbeater.cc:507] Master 127.10.87.62:36291 requested a full tablet report, sending...
I20260811 01:57:22.588732 10630 ts_manager.cc:194] Registered new tserver with Master: dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603)
I20260811 01:57:22.590781 10630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.87.1:34005
W20260811 01:57:22.688987 10836 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 01:57:22.689247 10836 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 01:57:22.689328 10836 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260811 01:57:22.692672 10836 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260811 01:57:22.692798 10836 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 01:57:22.692842 10836 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260811 01:57:22.692929 10836 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.87.2
I20260811 01:57:22.696918 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.87.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.87.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.87.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.10.87.62:36291
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.87.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 11 Aug 2026 01:43:15 UTC on bdcb31816ec0
build id 13280
I20260811 01:57:22.698266 10836 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 01:57:22.699540 10836 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260811 01:57:22.706172 10842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.706168 10846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.706323 10836 server_base.cc:1061] running on GCE node
W20260811 01:57:22.706172 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.706823 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 01:57:22.707391 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 01:57:22.708573 10836 hybrid_clock.cc:648] HybridClock initialized: now 1786413442708556 us; error 42 us; skew 500 ppm
I20260811 01:57:22.710495 10836 webserver.cc:533] Webserver started at http://127.10.87.2:44025/ using document root <none> and password file <none>
I20260811 01:57:22.711067 10836 fs_manager.cc:362] Metadata directory not provided
I20260811 01:57:22.711150 10836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 01:57:22.711376 10836 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 01:57:22.713147 10836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/instance:
uuid: "22a06c9f9ac940b68c5095c3834fd83b"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.713716 10836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal/instance:
uuid: "22a06c9f9ac940b68c5095c3834fd83b"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.717098 10836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 01:57:22.719295 10852 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.720300 10836 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260811 01:57:22.720432 10836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data,/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal
uuid: "22a06c9f9ac940b68c5095c3834fd83b"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.720530 10836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 01:57:22.758801 10836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260811 01:57:22.759528 10836 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 01:57:22.759734 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 01:57:22.760344 10836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260811 01:57:22.761445 10836 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 01:57:22.761516 10836 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.761590 10836 ts_tablet_manager.cc:616] Registered 0 tablets
I20260811 01:57:22.761654 10836 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260811 01:57:22.771195 10836 rpc_server.cc:307] RPC server started. Bound to: 127.10.87.2:38711
I20260811 01:57:22.771219 10977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.87.2:38711 every 8 connection(s)
I20260811 01:57:22.772212 10836 server_base.cc:1193] Dumped server information to /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb
I20260811 01:57:22.775858 10588 external_mini_cluster.cc:1434] Started /tmp/dist-test-task67xTv3/build/debug/bin/kudu as pid 10836
I20260811 01:57:22.775959 10588 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/wal/instance
I20260811 01:57:22.778278 10588 external_mini_cluster.cc:1372] Running /tmp/dist-test-task67xTv3/build/debug/bin/kudu
/tmp/dist-test-task67xTv3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.87.3:0
--local_ip_for_outbound_sockets=127.10.87.3
--webserver_interface=127.10.87.3
--webserver_port=0
--tserver_master_addrs=127.10.87.62:36291
--builtin_ntp_servers=127.10.87.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 01:57:22.781881 10978 heartbeater.cc:344] Connected to a master server at 127.10.87.62:36291
I20260811 01:57:22.782156 10978 heartbeater.cc:461] Registering TS with master...
I20260811 01:57:22.782723 10978 heartbeater.cc:507] Master 127.10.87.62:36291 requested a full tablet report, sending...
I20260811 01:57:22.783897 10630 ts_manager.cc:194] Registered new tserver with Master: 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711)
I20260811 01:57:22.784746 10630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.87.2:56303
W20260811 01:57:22.890535 10981 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 01:57:22.890758 10981 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 01:57:22.890808 10981 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260811 01:57:22.894186 10981 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260811 01:57:22.894305 10981 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 01:57:22.894335 10981 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260811 01:57:22.894416 10981 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.87.3
I20260811 01:57:22.898502 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.87.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.87.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.87.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.10.87.62:36291
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.87.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
build type DEBUG
built by None at 11 Aug 2026 01:43:15 UTC on bdcb31816ec0
build id 13280
I20260811 01:57:22.899623 10981 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 01:57:22.900748 10981 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260811 01:57:22.907374 10988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.907373 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 01:57:22.907382 10990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 01:57:22.907646 10981 server_base.cc:1061] running on GCE node
I20260811 01:57:22.908113 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 01:57:22.908749 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 01:57:22.909935 10981 hybrid_clock.cc:648] HybridClock initialized: now 1786413442909917 us; error 31 us; skew 500 ppm
I20260811 01:57:22.911883 10981 webserver.cc:533] Webserver started at http://127.10.87.3:33485/ using document root <none> and password file <none>
I20260811 01:57:22.912487 10981 fs_manager.cc:362] Metadata directory not provided
I20260811 01:57:22.912572 10981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 01:57:22.912814 10981 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 01:57:22.914480 10981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/instance:
uuid: "f4989bac1fb24983965dc41765af59fc"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.914976 10981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal/instance:
uuid: "f4989bac1fb24983965dc41765af59fc"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.918489 10981 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 01:57:22.920868 10997 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.921933 10981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 01:57:22.922062 10981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data,/tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal
uuid: "f4989bac1fb24983965dc41765af59fc"
format_stamp: "Formatted at 2026-08-11 01:57:22 on dist-test-slave-p3t5"
I20260811 01:57:22.922168 10981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 01:57:22.946276 10981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260811 01:57:22.947031 10981 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 01:57:22.947229 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 01:57:22.947837 10981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260811 01:57:22.948997 10981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 01:57:22.949065 10981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.949134 10981 ts_tablet_manager.cc:616] Registered 0 tablets
I20260811 01:57:22.949182 10981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 01:57:22.958577 10981 rpc_server.cc:307] RPC server started. Bound to: 127.10.87.3:39003
I20260811 01:57:22.958606 11122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.87.3:39003 every 8 connection(s)
I20260811 01:57:22.959610 10981 server_base.cc:1193] Dumped server information to /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/info.pb
I20260811 01:57:22.965904 10588 external_mini_cluster.cc:1434] Started /tmp/dist-test-task67xTv3/build/debug/bin/kudu as pid 10981
I20260811 01:57:22.966023 10588 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/wal/instance
I20260811 01:57:22.969236 11123 heartbeater.cc:344] Connected to a master server at 127.10.87.62:36291
I20260811 01:57:22.969513 11123 heartbeater.cc:461] Registering TS with master...
I20260811 01:57:22.970079 11123 heartbeater.cc:507] Master 127.10.87.62:36291 requested a full tablet report, sending...
I20260811 01:57:22.971155 10630 ts_manager.cc:194] Registered new tserver with Master: f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003)
I20260811 01:57:22.971711 10630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.87.3:38945
I20260811 01:57:22.981686 10588 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260811 01:57:22.995390 10630 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41784:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 01:57:22.999186 10630 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 01:57:23.019717 11047 tablet_service.cc:1511] Processing CreateTablet for tablet b0de48471b894bb8bbfaeca77de15585 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 01:57:23.019717 10761 tablet_service.cc:1511] Processing CreateTablet for tablet 0c339411098847e897c41c9b14d76167 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 01:57:23.019717 10904 tablet_service.cc:1511] Processing CreateTablet for tablet 0c339411098847e897c41c9b14d76167 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 01:57:23.020512 11050 tablet_service.cc:1511] Processing CreateTablet for tablet 0c339411098847e897c41c9b14d76167 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 01:57:23.020968 11048 tablet_service.cc:1511] Processing CreateTablet for tablet c7c13910df10490f88a4decb1d62b616 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 01:57:23.021087 10761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c339411098847e897c41c9b14d76167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.021098 11047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0de48471b894bb8bbfaeca77de15585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.021098 10904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c339411098847e897c41c9b14d76167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.021508 10758 tablet_service.cc:1511] Processing CreateTablet for tablet b0de48471b894bb8bbfaeca77de15585 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 01:57:23.021795 10758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0de48471b894bb8bbfaeca77de15585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.022965 10760 tablet_service.cc:1511] Processing CreateTablet for tablet c7c13910df10490f88a4decb1d62b616 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 01:57:23.023233 10760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c13910df10490f88a4decb1d62b616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.020053 10902 tablet_service.cc:1511] Processing CreateTablet for tablet b0de48471b894bb8bbfaeca77de15585 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 01:57:23.025885 10902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0de48471b894bb8bbfaeca77de15585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.019717 10903 tablet_service.cc:1511] Processing CreateTablet for tablet c7c13910df10490f88a4decb1d62b616 (DEFAULT_TABLE table=all-types-table [id=9157ca7217dd4e81aa37522cc3bf2dc4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 01:57:23.027769 10903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c13910df10490f88a4decb1d62b616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.029083 11142 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: Bootstrap starting.
I20260811 01:57:23.030059 11048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c13910df10490f88a4decb1d62b616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.031378 11050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c339411098847e897c41c9b14d76167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 01:57:23.031476 11142 tablet_bootstrap.cc:654] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.032512 11142 log.cc:826] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: Log is configured to *not* fsync() on all Append() calls
I20260811 01:57:23.033943 11141 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Bootstrap starting.
I20260811 01:57:23.034210 11143 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: Bootstrap starting.
I20260811 01:57:23.034546 11142 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: No bootstrap required, opened a new log
I20260811 01:57:23.034737 11142 ts_tablet_manager.cc:1403] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 01:57:23.036058 11141 tablet_bootstrap.cc:654] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.036686 11141 log.cc:826] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Log is configured to *not* fsync() on all Append() calls
I20260811 01:57:23.036799 11143 tablet_bootstrap.cc:654] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.037602 11143 log.cc:826] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: Log is configured to *not* fsync() on all Append() calls
I20260811 01:57:23.038435 11141 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: No bootstrap required, opened a new log
I20260811 01:57:23.038632 11141 ts_tablet_manager.cc:1403] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 01:57:23.038520 11142 raft_consensus.cc:359] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.038731 11142 raft_consensus.cc:385] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.038864 11142 raft_consensus.cc:740] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a06c9f9ac940b68c5095c3834fd83b, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.039335 11142 consensus_queue.cc:260] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.039441 11143 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: No bootstrap required, opened a new log
I20260811 01:57:23.039604 11143 ts_tablet_manager.cc:1403] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260811 01:57:23.040375 11142 ts_tablet_manager.cc:1434] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 01:57:23.040833 10978 heartbeater.cc:499] Master 127.10.87.62:36291 was elected leader, sending a full tablet report...
I20260811 01:57:23.040901 11142 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b: Bootstrap starting.
I20260811 01:57:23.042053 11142 tablet_bootstrap.cc:654] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.042141 11141 raft_consensus.cc:359] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.042397 11141 raft_consensus.cc:385] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.042511 11141 raft_consensus.cc:740] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.042968 11141 consensus_queue.cc:260] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.043562 10833 heartbeater.cc:499] Master 127.10.87.62:36291 was elected leader, sending a full tablet report...
I20260811 01:57:23.043776 11141 ts_tablet_manager.cc:1434] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 01:57:23.044070 11141 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7: Bootstrap starting.
I20260811 01:57:23.044747 11143 raft_consensus.cc:359] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.044977 11143 raft_consensus.cc:385] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.045082 11143 raft_consensus.cc:740] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4989bac1fb24983965dc41765af59fc, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.045149 11141 tablet_bootstrap.cc:654] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.045521 11143 consensus_queue.cc:260] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.046489 11143 ts_tablet_manager.cc:1434] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260811 01:57:23.046778 11141 tablet_bootstrap.cc:492] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7: No bootstrap required, opened a new log
I20260811 01:57:23.046888 11123 heartbeater.cc:499] Master 127.10.87.62:36291 was elected leader, sending a full tablet report...
I20260811 01:57:23.046941 11141 ts_tablet_manager.cc:1403] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 01:57:23.047665 11141 raft_consensus.cc:359] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.047813 11141 raft_consensus.cc:385] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.047863 11141 raft_consensus.cc:740] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.048077 11143 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: Bootstrap starting.
I20260811 01:57:23.048194 11141 consensus_queue.cc:260] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.048491 11141 ts_tablet_manager.cc:1434] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 01:57:23.048691 11141 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7: Bootstrap starting.
I20260811 01:57:23.049489 11143 tablet_bootstrap.cc:654] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.049803 11141 tablet_bootstrap.cc:654] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.050911 11143 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: No bootstrap required, opened a new log
I20260811 01:57:23.051007 11143 ts_tablet_manager.cc:1403] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260811 01:57:23.051012 11142 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b: No bootstrap required, opened a new log
I20260811 01:57:23.051118 11142 ts_tablet_manager.cc:1403] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260811 01:57:23.051581 11143 raft_consensus.cc:359] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.051717 11143 raft_consensus.cc:385] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.051760 11143 raft_consensus.cc:740] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4989bac1fb24983965dc41765af59fc, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.051762 11142 raft_consensus.cc:359] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.051884 11142 raft_consensus.cc:385] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.051916 11142 raft_consensus.cc:740] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a06c9f9ac940b68c5095c3834fd83b, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.051926 11143 consensus_queue.cc:260] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.052055 11142 consensus_queue.cc:260] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.052217 11143 ts_tablet_manager.cc:1434] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 01:57:23.052346 11142 ts_tablet_manager.cc:1434] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 01:57:23.052409 11143 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: Bootstrap starting.
I20260811 01:57:23.052529 11142 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: Bootstrap starting.
I20260811 01:57:23.053478 11143 tablet_bootstrap.cc:654] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.053727 11142 tablet_bootstrap.cc:654] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: Neither blocks nor log segments found. Creating new log.
I20260811 01:57:23.055151 11141 tablet_bootstrap.cc:492] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7: No bootstrap required, opened a new log
I20260811 01:57:23.055248 11141 ts_tablet_manager.cc:1403] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260811 01:57:23.055485 11142 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: No bootstrap required, opened a new log
I20260811 01:57:23.055498 11143 tablet_bootstrap.cc:492] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: No bootstrap required, opened a new log
I20260811 01:57:23.055564 11143 ts_tablet_manager.cc:1403] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 01:57:23.055564 11142 ts_tablet_manager.cc:1403] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 01:57:23.055799 11141 raft_consensus.cc:359] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.055900 11141 raft_consensus.cc:385] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.055950 11141 raft_consensus.cc:740] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.056013 11142 raft_consensus.cc:359] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.056013 11143 raft_consensus.cc:359] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.056116 11142 raft_consensus.cc:385] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.056116 11143 raft_consensus.cc:385] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 01:57:23.056097 11141 consensus_queue.cc:260] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.056151 11143 raft_consensus.cc:740] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4989bac1fb24983965dc41765af59fc, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.056151 11142 raft_consensus.cc:740] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a06c9f9ac940b68c5095c3834fd83b, State: Initialized, Role: FOLLOWER
I20260811 01:57:23.056300 11143 consensus_queue.cc:260] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.056303 11142 consensus_queue.cc:260] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.056468 11141 ts_tablet_manager.cc:1434] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 01:57:23.056561 11143 ts_tablet_manager.cc:1434] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 01:57:23.056561 11142 ts_tablet_manager.cc:1434] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260811 01:57:23.075788 10834 tablet.cc:2406] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 01:57:23.123687 11151 raft_consensus.cc:493] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 01:57:23.123916 11151 raft_consensus.cc:515] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.125284 11151 leader_election.cc:290] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711), f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003)
I20260811 01:57:23.130017 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0de48471b894bb8bbfaeca77de15585" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b" is_pre_election: true
I20260811 01:57:23.130220 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0de48471b894bb8bbfaeca77de15585" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc" is_pre_election: true
I20260811 01:57:23.130508 11074 raft_consensus.cc:2468] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.130507 10928 raft_consensus.cc:2468] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.131139 10712 leader_election.cc:304] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22a06c9f9ac940b68c5095c3834fd83b, dd70368ef7ad4752afbe694f0aef10b7, f4989bac1fb24983965dc41765af59fc; no voters:
I20260811 01:57:23.131446 11151 raft_consensus.cc:2804] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 01:57:23.131542 11151 raft_consensus.cc:493] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 01:57:23.131603 11151 raft_consensus.cc:3060] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.132674 11151 raft_consensus.cc:515] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.133068 11151 leader_election.cc:290] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Requested vote from peers 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711), f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003)
I20260811 01:57:23.133458 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0de48471b894bb8bbfaeca77de15585" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc"
I20260811 01:57:23.133476 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0de48471b894bb8bbfaeca77de15585" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b"
I20260811 01:57:23.133599 11074 raft_consensus.cc:3060] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.133608 10928 raft_consensus.cc:3060] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.135035 11074 raft_consensus.cc:2468] T b0de48471b894bb8bbfaeca77de15585 P f4989bac1fb24983965dc41765af59fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.135052 10928 raft_consensus.cc:2468] T b0de48471b894bb8bbfaeca77de15585 P 22a06c9f9ac940b68c5095c3834fd83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.135372 10712 leader_election.cc:304] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a06c9f9ac940b68c5095c3834fd83b, dd70368ef7ad4752afbe694f0aef10b7; no voters:
I20260811 01:57:23.135572 11151 raft_consensus.cc:2804] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 01:57:23.135818 11151 raft_consensus.cc:697] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 LEADER]: Becoming Leader. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Running, Role: LEADER
I20260811 01:57:23.136149 11151 consensus_queue.cc:237] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.140450 11151 raft_consensus.cc:493] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 01:57:23.140569 11151 raft_consensus.cc:515] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.141002 11151 leader_election.cc:290] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003), 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711)
I20260811 01:57:23.140606 10628 catalog_manager.cc:5719] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1). New cstate: current_term: 1 leader_uuid: "dd70368ef7ad4752afbe694f0aef10b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260811 01:57:23.141561 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c339411098847e897c41c9b14d76167" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b" is_pre_election: true
I20260811 01:57:23.141709 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c339411098847e897c41c9b14d76167" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc" is_pre_election: true
I20260811 01:57:23.141747 10928 raft_consensus.cc:2468] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.141846 11074 raft_consensus.cc:2468] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.142131 10712 leader_election.cc:304] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a06c9f9ac940b68c5095c3834fd83b, dd70368ef7ad4752afbe694f0aef10b7; no voters:
I20260811 01:57:23.142282 11151 raft_consensus.cc:2804] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 01:57:23.142333 11151 raft_consensus.cc:493] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 01:57:23.142367 11151 raft_consensus.cc:3060] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.143106 11151 raft_consensus.cc:515] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.143765 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c339411098847e897c41c9b14d76167" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc"
I20260811 01:57:23.143882 11074 raft_consensus.cc:3060] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.144178 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c339411098847e897c41c9b14d76167" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b"
I20260811 01:57:23.144295 10928 raft_consensus.cc:3060] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.144927 11074 raft_consensus.cc:2468] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.145272 10710 leader_election.cc:304] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd70368ef7ad4752afbe694f0aef10b7, f4989bac1fb24983965dc41765af59fc; no voters:
I20260811 01:57:23.145373 10928 raft_consensus.cc:2468] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.143631 11151 leader_election.cc:290] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Requested vote from peers f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003), 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711)
I20260811 01:57:23.145500 11162 raft_consensus.cc:2804] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 01:57:23.145660 11162 raft_consensus.cc:697] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 LEADER]: Becoming Leader. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Running, Role: LEADER
I20260811 01:57:23.145830 11162 consensus_queue.cc:237] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } }
I20260811 01:57:23.147722 10628 catalog_manager.cc:5719] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1). New cstate: current_term: 1 leader_uuid: "dd70368ef7ad4752afbe694f0aef10b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } health_report { overall_health: UNKNOWN } } }
W20260811 01:57:23.210652 11124 tablet.cc:2406] T 0c339411098847e897c41c9b14d76167 P f4989bac1fb24983965dc41765af59fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 01:57:23.257905 11162 raft_consensus.cc:493] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 01:57:23.258050 11162 raft_consensus.cc:515] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.258594 11162 leader_election.cc:290] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711), f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003)
I20260811 01:57:23.259136 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7c13910df10490f88a4decb1d62b616" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc" is_pre_election: true
I20260811 01:57:23.259162 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7c13910df10490f88a4decb1d62b616" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b" is_pre_election: true
I20260811 01:57:23.259317 11074 raft_consensus.cc:2468] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.259336 10928 raft_consensus.cc:2468] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 0.
I20260811 01:57:23.259672 10710 leader_election.cc:304] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd70368ef7ad4752afbe694f0aef10b7, f4989bac1fb24983965dc41765af59fc; no voters:
I20260811 01:57:23.259940 11162 raft_consensus.cc:2804] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 01:57:23.260013 11162 raft_consensus.cc:493] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 01:57:23.260089 11162 raft_consensus.cc:3060] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.261096 11162 raft_consensus.cc:515] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.261533 11162 leader_election.cc:290] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Requested vote from peers 22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711), f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003)
I20260811 01:57:23.261982 10928 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7c13910df10490f88a4decb1d62b616" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a06c9f9ac940b68c5095c3834fd83b"
I20260811 01:57:23.262022 11074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7c13910df10490f88a4decb1d62b616" candidate_uuid: "dd70368ef7ad4752afbe694f0aef10b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4989bac1fb24983965dc41765af59fc"
I20260811 01:57:23.262125 10928 raft_consensus.cc:3060] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.262162 11074 raft_consensus.cc:3060] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 0 FOLLOWER]: Advancing to term 1
I20260811 01:57:23.262952 10928 raft_consensus.cc:2468] T c7c13910df10490f88a4decb1d62b616 P 22a06c9f9ac940b68c5095c3834fd83b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.263010 11074 raft_consensus.cc:2468] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70368ef7ad4752afbe694f0aef10b7 in term 1.
I20260811 01:57:23.263329 10712 leader_election.cc:304] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a06c9f9ac940b68c5095c3834fd83b, dd70368ef7ad4752afbe694f0aef10b7; no voters:
I20260811 01:57:23.263497 11162 raft_consensus.cc:2804] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 01:57:23.263589 11162 raft_consensus.cc:697] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [term 1 LEADER]: Becoming Leader. State: Replica: dd70368ef7ad4752afbe694f0aef10b7, State: Running, Role: LEADER
I20260811 01:57:23.263737 11162 consensus_queue.cc:237] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } }
I20260811 01:57:23.265587 10628 catalog_manager.cc:5719] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1). New cstate: current_term: 1 leader_uuid: "dd70368ef7ad4752afbe694f0aef10b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd70368ef7ad4752afbe694f0aef10b7" member_type: VOTER last_known_addr { host: "127.10.87.1" port: 37603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
W20260811 01:57:23.275316 10979 tablet.cc:2406] T 0c339411098847e897c41c9b14d76167 P 22a06c9f9ac940b68c5095c3834fd83b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 01:57:23.586548 11162 consensus_queue.cc:1048] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:23.592784 11151 consensus_queue.cc:1048] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:23.606549 11151 consensus_queue.cc:1048] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:23.610113 11151 consensus_queue.cc:1048] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:23.689677 11151 consensus_queue.cc:1048] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a06c9f9ac940b68c5095c3834fd83b" member_type: VOTER last_known_addr { host: "127.10.87.2" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:23.693894 11151 consensus_queue.cc:1048] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4989bac1fb24983965dc41765af59fc" member_type: VOTER last_known_addr { host: "127.10.87.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 01:57:25.080245 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling FlushMRSOp(c7c13910df10490f88a4decb1d62b616): perf score=1.937278
W20260811 01:57:25.127667 11129 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.10.87.1:37603, user_credentials={real_user=slave}} blocked reactor thread for 60328.8us
I20260811 01:57:25.214982 11124 maintenance_manager.cc:419] P f4989bac1fb24983965dc41765af59fc: Scheduling FlushMRSOp(c7c13910df10490f88a4decb1d62b616): perf score=1.000000
I20260811 01:57:25.281876 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling FlushMRSOp(c7c13910df10490f88a4decb1d62b616): perf score=1.937278
W20260811 01:57:25.327544 11129 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.10.87.1:37603, user_credentials={real_user=slave}} blocked reactor thread for 70600.9us
W20260811 01:57:25.746100 11004 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-2/data/data (484941824 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 01:57:25.788635 11004 tablet_replica_mm_ops.cc:214] T c7c13910df10490f88a4decb1d62b616 P f4989bac1fb24983965dc41765af59fc: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet c7c13910df10490f88a4decb1d62b616: unable to open output file for column decimal128_val DECIMAL(38, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in c7c13910df10490f88a4decb1d62b616's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 01:57:25.788767 11004 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet c7c13910df10490f88a4decb1d62b616: unable to open output file for column decimal128_val DECIMAL(38, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in c7c13910df10490f88a4decb1d62b616's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fa77856cd5d google::LogMessage::Fail() at ??:0
@ 0x7fa778570b23 google::LogMessage::SendToLog() at ??:0
@ 0x7fa77856c75c google::LogMessage::Flush() at ??:0
@ 0x7fa77856dee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa77bff6b22 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fa779a8fcf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fa779a8ce08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7fa779a91139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c51949db9c std::function<>::operator()() at ??:0
@ 0x7fa779a4d767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa779a4e08f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa779a4f7db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c51949db9c std::function<>::operator()() at ??:0
@ 0x7fa779a3db52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa77a3bb6db start_thread at ??:0
@ 0x7fa77772361f clone at ??:0
I20260811 01:57:25.902755 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:25.908138 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
W20260811 01:57:26.167308 10710 connection.cc:573] client connection to 127.10.87.3:39003 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 01:57:26.167505 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 01:57:26.168808 11212 negotiation.cc:336] Failed RPC negotiation. Trace:
0811 01:57:25.902077 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.10.87.3:39003 (local address 127.0.0.1:41070)
0811 01:57:25.902314 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 01:57:25.902331 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0811 01:57:25.902409 (+ 78us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0811 01:57:26.168316 (+265907us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155}
W20260811 01:57:26.168892 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:26.178818 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:26.188481 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 01:57:26.286412 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: FlushMRSOp(c7c13910df10490f88a4decb1d62b616) complete. Timing: real 1.206s user 0.584s sys 0.008s Metrics: {"bytes_written":472622,"cfile_init":1,"compiler_manager_pool.queue_time_us":13087,"compiler_manager_pool.run_cpu_time_us":323331,"compiler_manager_pool.run_wall_time_us":413332,"dirs.queue_time_us":219,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":935,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2916,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":345,"threads_started":3,"wal-append.queue_time_us":246}
I20260811 01:57:26.287914 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling UndoDeltaBlockGCOp(c7c13910df10490f88a4decb1d62b616): 62922 bytes on disk
I20260811 01:57:26.288594 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: UndoDeltaBlockGCOp(c7c13910df10490f88a4decb1d62b616) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20260811 01:57:26.289127 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling FlushMRSOp(0c339411098847e897c41c9b14d76167): perf score=1.937278
I20260811 01:57:26.339071 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: FlushMRSOp(c7c13910df10490f88a4decb1d62b616) complete. Timing: real 1.057s user 0.615s sys 0.032s Metrics: {"bytes_written":472622,"cfile_init":1,"compiler_manager_pool.queue_time_us":8084,"compiler_manager_pool.run_cpu_time_us":324811,"compiler_manager_pool.run_wall_time_us":438449,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":816,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3027,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":229120,"thread_start_us":367,"threads_started":3,"wal-append.queue_time_us":212}
I20260811 01:57:26.340310 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling UndoDeltaBlockGCOp(c7c13910df10490f88a4decb1d62b616): 62922 bytes on disk
I20260811 01:57:26.341017 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: UndoDeltaBlockGCOp(c7c13910df10490f88a4decb1d62b616) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260811 01:57:26.341446 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling FlushMRSOp(0c339411098847e897c41c9b14d76167): perf score=1.937278
I20260811 01:57:26.394940 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:26.395651 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
W20260811 01:57:26.409947 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:26.428319 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:26.671241 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: FlushMRSOp(0c339411098847e897c41c9b14d76167) complete. Timing: real 0.382s user 0.376s sys 0.001s Metrics: {"bytes_written":436486,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2331,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":468}
I20260811 01:57:26.671928 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling FlushMRSOp(b0de48471b894bb8bbfaeca77de15585): perf score=1.937278
I20260811 01:57:26.731640 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:26.742014 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: FlushMRSOp(0c339411098847e897c41c9b14d76167) complete. Timing: real 0.400s user 0.398s sys 0.000s Metrics: {"bytes_written":436486,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1431,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2839,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260811 01:57:26.742844 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling UndoDeltaBlockGCOp(0c339411098847e897c41c9b14d76167): 62922 bytes on disk
I20260811 01:57:26.743393 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: UndoDeltaBlockGCOp(0c339411098847e897c41c9b14d76167) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20260811 01:57:26.743985 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling FlushMRSOp(b0de48471b894bb8bbfaeca77de15585): perf score=1.937278
I20260811 01:57:26.748143 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:26.777858 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:27.061717 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: FlushMRSOp(b0de48471b894bb8bbfaeca77de15585) complete. Timing: real 0.390s user 0.381s sys 0.002s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2702,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":224}
I20260811 01:57:27.062322 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling UndoDeltaBlockGCOp(b0de48471b894bb8bbfaeca77de15585): 63218 bytes on disk
I20260811 01:57:27.062752 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: UndoDeltaBlockGCOp(b0de48471b894bb8bbfaeca77de15585) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260811 01:57:27.063223 10834 maintenance_manager.cc:419] P dd70368ef7ad4752afbe694f0aef10b7: Scheduling UndoDeltaBlockGCOp(0c339411098847e897c41c9b14d76167): 62922 bytes on disk
I20260811 01:57:27.063573 10713 maintenance_manager.cc:643] P dd70368ef7ad4752afbe694f0aef10b7: UndoDeltaBlockGCOp(0c339411098847e897c41c9b14d76167) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20260811 01:57:27.100754 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: FlushMRSOp(b0de48471b894bb8bbfaeca77de15585) complete. Timing: real 0.357s user 0.345s sys 0.008s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1183,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2670,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":288}
I20260811 01:57:27.101473 10979 maintenance_manager.cc:419] P 22a06c9f9ac940b68c5095c3834fd83b: Scheduling UndoDeltaBlockGCOp(b0de48471b894bb8bbfaeca77de15585): 63218 bytes on disk
I20260811 01:57:27.101963 10860 maintenance_manager.cc:643] P 22a06c9f9ac940b68c5095c3834fd83b: UndoDeltaBlockGCOp(b0de48471b894bb8bbfaeca77de15585) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260811 01:57:27.170665 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:27.170719 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:27.190634 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:27.755419 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:27.755676 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:27.776808 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:28.346077 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260811 01:57:28.501914 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 01:57:28.606729 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:28.607411 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:28.633214 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:28.945935 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 01:57:29.687428 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:29.687494 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:29.707878 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:30.707964 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260811 01:57:30.761840 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:30.762152 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:30.784538 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:30.938627 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260811 01:57:31.406179 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 01:57:31.533389 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260811 01:57:31.841331 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:31.841707 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:31.868280 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:32.931103 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:32.931420 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:32.952378 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:33.078591 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260811 01:57:33.598398 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260811 01:57:33.939314 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260811 01:57:34.007290 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:34.008234 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:34.028051 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:35.086501 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:35.086536 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:35.106029 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:35.206547 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260811 01:57:35.919083 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 01:57:36.159615 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:36.159940 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:36.180430 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:36.443954 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:57:36.537039 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 01:57:37.236657 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:37.236728 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:37.257756 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:37.844098 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 01:57:38.319891 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:38.320142 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
W20260811 01:57:38.336544 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:38.348567 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:39.214056 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 01:57:39.402518 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:39.403203 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:39.426126 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:40.192852 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 01:57:40.480561 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:40.480626 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:40.503655 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:40.777336 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 01:57:41.557652 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:41.557725 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:41.578130 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:41.716056 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:57:41.716856 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 01:57:42.632519 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:42.632519 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:42.657655 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:42.879235 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260811 01:57:43.464192 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 01:57:43.715190 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:43.715497 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:43.741225 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:44.094589 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 01:57:44.796202 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:44.796567 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:44.822923 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:45.476351 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260811 01:57:45.883919 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:45.883940 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:45.906184 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:45.909719 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260811 01:57:46.537844 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260811 01:57:46.971508 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:46.971555 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:47.002170 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:47.009575 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:57:47.936195 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 01:57:48.058739 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:48.058739 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:48.083968 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:48.500037 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260811 01:57:48.907979 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 01:57:49.145325 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:49.146329 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:49.167100 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:57:50.219632 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:50.219961 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:50.240125 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:50.301767 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 01:57:51.275066 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260811 01:57:51.297045 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:51.297528 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:51.318866 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:51.481513 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 01:57:52.255609 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20260811 01:57:52.373194 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:52.373231 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:52.393352 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:52.708228 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 01:57:53.447455 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:53.448546 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:53.471801 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:53.522668 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260811 01:57:53.978073 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 01:57:54.530794 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:54.530817 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:54.551638 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:55.274014 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 01:57:55.606006 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:55.606922 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:55.628665 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:55.808240 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260811 01:57:56.586927 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 01:57:56.685593 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:56.685714 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:56.710098 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:57.475318 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 01:57:57.515005 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260811 01:57:57.763273 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:57.763724 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:57.787168 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:58.022143 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260811 01:57:58.843127 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:57:58.843142 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:58.872097 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:57:59.062878 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260811 01:57:59.829432 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 01:57:59.932006 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:57:59.933190 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:57:59.954684 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:00.178293 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 01:58:01.011132 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:01.011268 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:01.031454 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:01.641645 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 01:58:02.088261 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:02.088292 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:02.111864 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:02.222900 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260811 01:58:02.580370 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 01:58:02.581274 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 01:58:03.167423 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:03.167418 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:03.189157 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:04.219646 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 01:58:04.243568 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:04.244053 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:04.265538 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:04.691860 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 01:58:05.111923 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260811 01:58:05.324926 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:05.325122 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:05.347889 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:06.402410 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:06.402469 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:06.427965 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:06.656359 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 01:58:07.271813 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 01:58:07.483672 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:07.483929 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:07.507531 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:07.682652 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:58:07.840255 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 01:58:08.561990 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:08.562160 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:08.589365 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:09.009302 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260811 01:58:09.520411 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 01:58:09.642693 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:09.642887 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:09.664654 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:10.554903 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 01:58:10.723145 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:10.723213 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:10.750180 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:11.539342 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 01:58:11.806280 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:11.806280 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:11.826370 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:11.902040 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 01:58:12.823086 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20260811 01:58:12.880534 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:12.880771 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:12.906272 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:12.992363 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 01:58:13.925787 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260811 01:58:13.964833 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:13.964900 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:13.986277 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:14.308971 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 01:58:15.040287 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:15.040637 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:15.062391 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:15.486155 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 01:58:16.122287 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:16.122339 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:16.148643 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:16.270238 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260811 01:58:16.464200 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 01:58:17.204160 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:17.204408 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:17.231933 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:17.821877 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260811 01:58:18.112802 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20260811 01:58:18.286026 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:18.286386 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:18.307663 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:18.670856 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260811 01:58:19.042991 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260811 01:58:19.362466 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:19.362738 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:19.388375 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:20.443423 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:20.443472 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:20.464840 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:20.543627 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260811 01:58:21.286681 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260811 01:58:21.524394 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:21.524427 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:21.548285 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:21.761777 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260811 01:58:22.608346 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:22.608372 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:22.629383 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:23.144524 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 01:58:23.145246 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260811 01:58:23.685170 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:23.685392 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:23.707369 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:23.752352 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 01:58:24.359349 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 01:58:24.763384 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:24.763442 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:24.785745 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:25.701088 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 01:58:25.841148 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:25.841518 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:25.863757 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:25.962815 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260811 01:58:26.773062 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260811 01:58:26.927549 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:26.927935 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:26.953752 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:28.009280 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:28.009280 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:28.035368 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:28.187734 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:58:28.188532 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260811 01:58:28.292599 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260811 01:58:29.089655 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:29.089733 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:29.110692 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:29.203617 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260811 01:58:30.164942 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:30.165292 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:30.185524 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:30.676411 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260811 01:58:30.831310 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260811 01:58:31.241369 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:31.241355 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:31.263391 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:31.406029 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 01:58:32.322602 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:32.332155 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:32.354274 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:33.225765 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:58:33.226550 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260811 01:58:33.273595 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 01:58:33.438006 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:33.438006 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:33.484120 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:33.875262 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260811 01:58:34.540745 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:34.541203 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:34.568653 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:35.625173 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:35.625823 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:35.649843 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:35.698846 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260811 01:58:35.892325 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260811 01:58:36.691009 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260811 01:58:36.704414 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:36.704491 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:36.725155 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:37.781298 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:37.781311 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:37.809584 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:37.969393 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 01:58:38.548887 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:58:38.647658 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260811 01:58:38.863387 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:38.864377 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:38.888718 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:39.149413 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260811 01:58:39.941871 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:39.941901 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:39.968436 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:40.536012 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260811 01:58:41.023962 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:41.023962 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:41.045027 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:41.090667 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260811 01:58:41.307430 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 01:58:42.098553 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:42.098605 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:42.124141 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:42.962041 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 01:58:43.181320 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:43.181823 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:43.203142 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:43.506302 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260811 01:58:43.816782 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 01:58:43.817607 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 01:58:44.256848 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:44.258740 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:44.279994 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:45.308701 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 01:58:45.336107 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:45.336494 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:45.357841 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:45.905346 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260811 01:58:46.246762 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 01:58:46.414086 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:46.414093 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:46.439790 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:47.492239 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:47.492528 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:47.516503 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:47.702755 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260811 01:58:48.028417 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 01:58:48.577545 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:48.577862 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:48.603659 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:48.623351 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 01:58:48.906657 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20260811 01:58:49.657502 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:49.657502 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:49.684234 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:50.261677 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 01:58:50.262208 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260811 01:58:50.736955 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:50.736956 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:50.759693 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:51.188426 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 01:58:51.813150 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:51.813511 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:51.837303 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:52.606841 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260811 01:58:52.823473 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 01:58:52.891217 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:52.891290 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:52.916637 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:58:53.975629 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:53.975994 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:54.003420 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:54.015646 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 01:58:54.016412 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260811 01:58:55.020486 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 01:58:55.057242 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:55.057291 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:55.080432 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:55.430301 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 01:58:56.135026 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:56.135269 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:56.155665 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:56.281136 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260811 01:58:57.211015 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:57.211124 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:57.235236 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:57.440017 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260811 01:58:58.048699 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260811 01:58:58.289229 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:58:58.289295 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:58.310019 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:58:58.836179 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260811 01:58:59.284333 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20260811 01:58:59.371287 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:58:59.371343 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:58:59.397948 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:00.059584 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260811 01:59:00.369803 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 01:59:00.455535 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:00.456019 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:00.480980 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:01.043562 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 01:59:01.535346 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:01.536458 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:01.557756 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:59:02.615105 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:02.615226 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:02.639807 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:02.643369 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260811 01:59:02.684731 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 01:59:03.696386 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:03.697284 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:03.725315 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:03.842490 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260811 01:59:04.321367 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20260811 01:59:04.783814 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:04.784157 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:04.808104 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:04.865828 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260811 01:59:05.370314 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260811 01:59:05.864655 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:05.865103 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:05.885881 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:06.486578 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 01:59:06.940106 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:06.940102 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:06.965526 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:07.383669 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260811 01:59:07.724210 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 01:59:08.024055 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:08.024091 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:08.045295 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:08.868824 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260811 01:59:09.104194 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:09.104194 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:09.131851 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:09.491946 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:59:09.976627 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 01:59:10.187251 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260811 01:59:10.190698 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:10.190698 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:10.217296 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:59:11.270246 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:11.270572 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:11.296574 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:11.602934 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260811 01:59:12.134605 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 01:59:12.351820 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:12.352370 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:12.373698 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:12.699137 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 01:59:13.429111 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:13.429121 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:13.453431 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:14.130438 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 01:59:14.509577 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:14.509594 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:14.529376 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:14.634881 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:59:14.835645 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260811 01:59:15.147594 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 01:59:15.587639 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:15.587965 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:15.611378 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:59:16.671913 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:16.671977 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
W20260811 01:59:16.681952 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:16.693872 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:17.278786 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260811 01:59:17.379375 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260811 01:59:17.748910 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:17.749439 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:17.769964 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 01:59:18.824779 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:18.824772 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:18.851351 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:18.979666 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 01:59:19.776353 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 01:59:19.777170 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260811 01:59:19.908304 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:19.908319 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.1:37603, tserver@127.10.87.2:38711
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:19.930327 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:20.018448 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260811 01:59:20.988505 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260811 01:59:20.988588 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:21.009039 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:21.456355 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20260811 01:59:22.064721 10904 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260811 01:59:22.064754 10760 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b6613112d9004b319b2c743bfe4067e9 | 127.10.87.62:36291 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.10.87.20:32997 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
22a06c9f9ac940b68c5095c3834fd83b | 127.10.87.2:38711 | HEALTHY | <none> | 0 | 0
dd70368ef7ad4752afbe694f0aef10b7 | 127.10.87.1:37603 | HEALTHY | <none> | 3 | 0
f4989bac1fb24983965dc41765af59fc | 127.10.87.3:39003 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.10.87.3:39003: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.10.87.2:38711, 127.10.87.1:37603
ipki_server_key_size | 768 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.1 | experimental | 127.10.87.1:37603
local_ip_for_outbound_sockets | 127.10.87.2 | experimental | 127.10.87.2:38711
never_fsync | true | unsafe,advanced | 127.10.87.2:38711, 127.10.87.1:37603
openssl_security_level_override | 0 | unsafe,hidden | 127.10.87.2:38711, 127.10.87.1:37603
rpc_server_allow_ephemeral_ports | true | unsafe | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_format | pb | hidden | 127.10.87.2:38711, 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-0/data/info.pb | hidden | 127.10.87.1:37603
server_dump_info_path | /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/ts-1/data/info.pb | hidden | 127.10.87.2:38711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.10.87.2:38711, 127.10.87.1:37603
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.10.87.20:32997 | 127.10.87.2:38711, 127.10.87.1:37603
time_source | builtin | 127.10.87.2:38711, 127.10.87.1:37603
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.19.0-SNAPSHOT | master@127.10.87.62:36291, tserver@127.10.87.2:38711, tserver@127.10.87.1:37603
Tablet Summary
Tablet 0c339411098847e897c41c9b14d76167 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = f4989bac1fb24983965dc41765af59fc
C = 22a06c9f9ac940b68c5095c3834fd83b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c7c13910df10490f88a4decb1d62b616 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b0de48471b894bb8bbfaeca77de15585 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
dd70368ef7ad4752afbe694f0aef10b7 (127.10.87.1:37603): RUNNING [LEADER]
22a06c9f9ac940b68c5095c3834fd83b (127.10.87.2:38711): RUNNING
f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): TS unavailable
All reported replicas are:
A = dd70368ef7ad4752afbe694f0aef10b7
B = 22a06c9f9ac940b68c5095c3834fd83b
C = f4989bac1fb24983965dc41765af59fc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 01:59:22.090428 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 01:59:22.452486 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20260811 01:59:22.484176 10618 maintenance_manager.cc:419] P b6613112d9004b319b2c743bfe4067e9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371
W20260811 01:59:22.486486 10617 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-task67xTv3/test-tmp/all_types-itest.4.AllTypesItest_2.TestAllKeyTypes.1786413442070788-10588-0/minicluster-data/master-0/data/data (480206848 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 01:59:22.487147 10617 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P b6613112d9004b319b2c743bfe4067e9: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 01:59:22.487217 10617 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f82d1246d5d google::LogMessage::Fail() at ??:0
@ 0x7f82d124ab23 google::LogMessage::SendToLog() at ??:0
@ 0x7f82d124675c google::LogMessage::Flush() at ??:0
@ 0x7f82d1247ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f82d4cd0b22 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f82d2769cf2 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f82d2766e08 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f82d276b139 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56500537db9c std::function<>::operator()() at ??:0
@ 0x7f82d2727767 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f82d272808f _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f82d27297db _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56500537db9c std::function<>::operator()() at ??:0
@ 0x7f82d2717b52 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f82d30956db start_thread at ??:0
@ 0x7f82d03fd61f clone at ??:0
W20260811 01:59:22.593704 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 01:59:22.936960 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 01:59:23.261253 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 01:59:24.020303 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 01:59:25.005935 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 01:59:25.110255 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 01:59:25.192685 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 01:59:26.450925 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 01:59:27.826690 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 01:59:27.853206 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 01:59:28.604202 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 01:59:30.069867 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:59:30.217861 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 01:59:30.424713 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 01:59:31.220059 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 01:59:32.564669 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 01:59:33.075884 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 01:59:33.588131 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 01:59:35.102756 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 01:59:35.103538 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 01:59:35.508222 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 01:59:35.989602 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 01:59:37.806234 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 01:59:38.226378 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 01:59:38.667900 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 01:59:40.259740 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 01:59:40.481271 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 01:59:40.850080 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 01:59:41.025996 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 01:59:42.963618 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 01:59:43.190021 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 01:59:43.641134 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 01:59:45.446614 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 01:59:45.447508 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 01:59:45.744808 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 01:59:46.140233 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 01:59:48.153493 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 01:59:48.212875 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 01:59:48.681206 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 01:59:50.457147 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 01:59:50.458010 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 01:59:50.919878 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 01:59:51.189693 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 01:59:52.971784 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 01:59:53.438369 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 01:59:53.586174 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 01:59:55.547659 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 01:59:55.567319 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 01:59:55.999547 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 01:59:56.073859 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 01:59:58.101858 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 01:59:58.541606 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 01:59:58.703613 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 02:00:00.556869 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:00:00.558130 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 02:00:01.153724 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 02:00:01.299230 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 02:00:02.902475 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 02:00:03.656913 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 02:00:03.714102 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 02:00:05.375423 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 02:00:05.598733 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:06.066390 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 02:00:06.196029 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 02:00:07.963912 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 02:00:08.547421 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 02:00:08.635182 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 02:00:10.427292 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 02:00:10.625020 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:00:11.201220 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 02:00:11.415771 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 02:00:12.755894 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 02:00:13.570384 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 02:00:13.748170 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 02:00:15.122762 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 02:00:15.718343 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:16.196328 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 02:00:16.493705 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 02:00:17.785210 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 02:00:18.804912 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 02:00:18.805867 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 02:00:20.493415 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 02:00:20.832553 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:00:21.295001 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 02:00:21.457458 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 02:00:23.019394 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (60 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:00:23.101770 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:00:23.231640 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:00:23.337859 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (60 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:00:23.679433 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 02:00:23.936666 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 02:00:25.593587 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 02:00:25.861693 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:26.348347 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 02:00:26.407558 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 02:00:28.342514 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 02:00:28.628358 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 02:00:28.706331 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 02:00:30.813426 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 02:00:31.208173 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:31.209030 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 02:00:31.262791 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 02:00:33.350184 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 02:00:33.537932 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 02:00:33.899169 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 02:00:35.876320 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 02:00:35.982203 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 02:00:36.400669 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:36.594324 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 02:00:38.162804 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 02:00:38.544946 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 02:00:39.009877 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 02:00:40.735497 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 02:00:40.971350 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 02:00:41.346554 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 02:00:41.490243 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:00:43.175668 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 02:00:43.420801 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 02:00:43.774523 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 02:00:45.767452 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 02:00:45.828979 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 02:00:46.296519 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 02:00:46.891175 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:48.426620 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 02:00:48.515096 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 02:00:48.979638 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 02:00:50.870076 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 02:00:51.049028 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 02:00:51.505270 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 02:00:51.903946 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:00:53.324368 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 02:00:53.642791 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 02:00:53.741444 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 02:00:55.695176 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 02:00:56.345487 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 02:00:56.531086 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 02:00:57.102790 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:00:58.168519 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 02:00:58.792349 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 02:00:59.243074 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 02:01:00.769520 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 02:01:01.056708 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 02:01:01.875187 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 02:01:02.327065 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:01:03.386418 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 02:01:03.605043 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 02:01:04.156879 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 02:01:05.839524 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 02:01:06.229949 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 02:01:06.546980 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 02:01:07.329823 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:01:08.373083 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 02:01:08.752427 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 02:01:08.968734 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 02:01:11.094596 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 02:01:11.177565 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 02:01:11.534935 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 02:01:12.522483 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:01:13.606666 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 02:01:13.693189 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 02:01:14.153190 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 02:01:15.924527 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 02:01:16.192662 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 02:01:16.400691 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 02:01:17.579784 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:01:18.407148 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 02:01:18.537366 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 02:01:19.145138 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 02:01:20.728466 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 02:01:20.961041 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 02:01:21.897730 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 02:01:22.755268 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:01:23.103420 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (120 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:01:23.209911 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 02:01:23.410772 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (120 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:01:23.739876 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:01:24.351723 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:01:24.702909 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 02:01:25.743846 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 02:01:26.196889 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 02:01:27.256062 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 02:01:27.804621 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:01:28.397518 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 02:01:28.702687 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 02:01:29.758708 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 02:01:30.931348 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 02:01:31.144459 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 02:01:32.133745 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 02:01:32.820246 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:01:33.440204 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 02:01:33.895993 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 02:01:34.926025 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 02:01:35.768658 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 02:01:36.140519 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 02:01:37.198747 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 02:01:37.925768 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:01:38.048385 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 02:01:38.371222 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 02:01:39.584723 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 02:01:40.502399 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 02:01:40.663436 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 02:01:42.051050 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 02:01:43.031119 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:01:43.031903 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 02:01:43.130111 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 02:01:44.381822 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 02:01:45.208655 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 02:01:45.566807 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 02:01:47.114061 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 02:01:47.477299 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 02:01:47.847071 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 02:01:48.052449 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:01:49.631023 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 02:01:49.699489 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 02:01:50.484234 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 02:01:52.264607 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 02:01:52.361739 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 02:01:52.822036 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 02:01:53.280815 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:01:54.683202 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 02:01:55.014276 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 02:01:55.191027 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 02:01:57.185741 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 02:01:57.564869 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 02:01:57.651806 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 02:01:58.532284 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:01:59.512468 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 02:02:00.095371 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 02:02:00.215654 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 02:02:01.947268 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 02:02:02.729034 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 02:02:02.930963 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 02:02:03.705303 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:02:04.148777 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 02:02:05.045171 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 02:02:05.443856 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 02:02:06.831012 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 02:02:07.168336 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 02:02:07.931241 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 02:02:08.721679 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:02:09.091945 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 02:02:09.783140 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 02:02:10.616142 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 02:02:11.659368 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 02:02:12.474820 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 02:02:13.312019 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 02:02:13.932062 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:02:14.007974 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 02:02:15.132663 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 02:02:16.076807 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 02:02:16.649828 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 02:02:17.625696 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 02:02:18.159732 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 02:02:19.085098 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:02:19.105123 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 02:02:20.063136 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260811 02:02:20.817124 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 02:02:21.389542 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 02:02:22.498770 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 02:02:23.187544 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (180 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:02:23.296993 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260811 02:02:23.485100 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (180 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:02:23.858639 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260811 02:02:24.236025 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:02:24.803942 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:02:25.527143 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
I20260811 02:02:25.839167 14787 consensus_queue.cc:579] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Leader has been unable to successfully communicate with peer f4989bac1fb24983965dc41765af59fc for more than 300 seconds (300.060s)
W20260811 02:02:25.840919 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20260811 02:02:25.876740 14799 consensus_queue.cc:579] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Leader has been unable to successfully communicate with peer f4989bac1fb24983965dc41765af59fc for more than 300 seconds (300.051s)
I20260811 02:02:25.950102 14799 consensus_queue.cc:579] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 [LEADER]: Leader has been unable to successfully communicate with peer f4989bac1fb24983965dc41765af59fc for more than 300 seconds (300.148s)
W20260811 02:02:26.354916 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 02:02:27.412978 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 02:02:28.349500 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 02:02:28.970734 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 02:02:29.591576 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:02:30.156242 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 02:02:30.857478 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 02:02:31.551986 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 02:02:32.660063 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 02:02:33.426772 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 02:02:33.915781 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 02:02:34.840217 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:02:34.933480 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 02:02:35.924563 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 02:02:36.303623 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 02:02:37.393126 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 02:02:38.480172 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 02:02:38.762764 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 02:02:40.041419 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:02:40.053318 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 02:02:40.707427 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 02:02:41.199021 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 02:02:42.803306 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 02:02:43.405551 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 02:02:43.759630 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 02:02:45.250586 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:02:45.251370 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 02:02:45.908231 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 02:02:46.475240 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 02:02:47.673353 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 02:02:48.550143 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 02:02:49.232393 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 02:02:49.975068 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 02:02:50.327435 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:02:51.308495 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 02:02:51.720587 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 02:02:52.526180 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 02:02:53.716236 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 02:02:54.039929 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 02:02:55.054698 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 02:02:55.546350 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:02:56.218828 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 02:02:56.421763 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 02:02:57.614873 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 02:02:58.839859 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 02:02:59.322301 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 02:03:00.047796 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 02:03:00.627478 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:01.402789 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 02:03:02.055996 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 02:03:02.584183 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 02:03:03.902335 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 02:03:04.454679 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 02:03:04.788628 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 02:03:05.652523 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:03:06.566239 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 02:03:07.051522 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 02:03:07.065094 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 02:03:09.238032 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 02:03:09.433154 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 02:03:09.546927 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 02:03:10.715875 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:11.670161 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 02:03:11.815508 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 02:03:11.831498 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 02:03:14.044893 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 02:03:14.220906 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 02:03:14.409456 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 02:03:15.973014 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:03:16.411960 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 02:03:16.441550 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 02:03:17.058498 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 02:03:18.846073 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 02:03:19.082543 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 02:03:19.808493 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 02:03:21.177142 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:03:21.416294 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 02:03:21.502542 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 02:03:22.194242 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 02:03:23.273135 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (240 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:03:23.886977 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 02:03:24.025489 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (243 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:03:24.101735 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 02:03:24.884605 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 02:03:26.385914 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:26.386729 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:03:26.695235 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:03:26.701431 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 02:03:27.596515 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 02:03:28.580581 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 02:03:28.985071 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 02:03:30.342379 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 02:03:31.163478 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 02:03:31.548154 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:03:31.695811 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 02:03:32.692659 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 02:03:33.631697 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 02:03:34.134272 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 02:03:35.259743 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 02:03:36.053098 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 02:03:36.538271 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 02:03:36.649654 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:03:37.717432 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 02:03:38.906035 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 02:03:38.971532 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 02:03:40.289316 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 02:03:41.237332 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 02:03:41.549268 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 02:03:41.743253 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:42.740530 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 02:03:43.652980 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 02:03:43.989214 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 02:03:45.448668 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 02:03:46.157217 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 02:03:46.221554 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 02:03:46.763497 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:03:48.205596 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 02:03:48.567531 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 02:03:48.603518 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 02:03:50.642429 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 02:03:51.130246 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 02:03:51.209466 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 02:03:51.770093 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:52.771314 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 02:03:53.576138 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 02:03:53.810696 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 02:03:55.245229 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 02:03:56.255721 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 02:03:56.450343 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 02:03:56.850754 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:03:57.545277 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 02:03:58.755985 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 02:03:58.756671 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 02:03:59.768828 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 02:04:01.041561 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 02:04:01.082718 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 02:04:01.899505 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:04:02.011013 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 02:04:03.625610 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 02:04:03.707736 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 02:04:04.329650 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 02:04:06.388123 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 02:04:06.541714 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 02:04:06.665272 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 02:04:06.923148 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:04:08.819566 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 02:04:09.083493 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 02:04:09.134624 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 02:04:11.400962 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 02:04:11.519593 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 02:04:11.760617 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 02:04:12.123857 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:04:14.238859 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 02:04:14.242643 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 02:04:14.292879 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 02:04:16.772939 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 02:04:16.845165 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 02:04:16.920933 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 02:04:17.293731 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:04:19.366380 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 02:04:19.389529 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 02:04:19.592242 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 02:04:21.719086 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 02:04:21.800510 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 02:04:21.823717 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 02:04:22.388468 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:04:23.358160 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (300 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:04:24.099946 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (303 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:04:24.181450 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 02:04:24.261735 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 02:04:24.370174 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 02:04:26.779269 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 02:04:26.919031 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 02:04:26.972467 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 02:04:27.405172 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:04:27.822942 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:04:29.238054 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 02:04:29.341589 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 02:04:29.662997 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 02:04:31.737703 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 02:04:31.755517 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 02:04:32.259753 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 02:04:32.515901 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:04:34.047243 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 02:04:34.104632 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 02:04:34.557859 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 02:04:36.511328 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 02:04:36.793432 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 02:04:37.308221 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 02:04:37.749064 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:04:39.064186 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 02:04:39.556273 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 02:04:39.715176 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 02:04:41.760907 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 02:04:42.096139 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 02:04:42.156471 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 02:04:43.119491 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:04:44.261889 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 02:04:44.420266 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 02:04:44.641265 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 02:04:46.950455 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 02:04:46.999591 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 02:04:47.177652 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 02:04:48.465723 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:04:49.384642 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 02:04:49.557478 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 02:04:49.816551 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 02:04:51.533722 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 02:04:51.927973 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 02:04:52.269104 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 02:04:53.556917 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:04:54.024010 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 02:04:54.412710 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 02:04:54.758217 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 02:04:56.376680 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 02:04:56.961413 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 02:04:57.228755 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 02:04:58.590631 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:04:58.839015 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 02:04:59.615969 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 02:04:59.686398 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 02:05:01.325754 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 02:05:02.182368 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 02:05:02.192960 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 02:05:03.448594 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 02:05:03.632637 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:05:04.811398 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 02:05:04.930462 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 02:05:06.008662 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 02:05:07.335791 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 02:05:07.742580 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 02:05:08.430305 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 02:05:08.920441 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:05:10.022856 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 02:05:10.189283 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 02:05:10.912236 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 02:05:12.419970 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 02:05:12.663021 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 02:05:13.383680 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 02:05:13.990971 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:14.978111 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 02:05:15.043382 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 02:05:16.199790 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 02:05:17.425163 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 02:05:17.686049 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 02:05:18.703240 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 02:05:19.155042 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:05:19.736763 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 02:05:20.029807 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 02:05:21.268801 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 02:05:22.334901 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 02:05:22.375579 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 02:05:23.443801 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (360 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:05:23.631982 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 02:05:24.164310 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:24.173280 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (363 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:05:24.824965 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 02:05:24.916311 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 02:05:26.293583 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 02:05:27.374222 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 02:05:27.407471 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 02:05:28.740772 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:05:28.960883 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:05:29.342813 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:29.672122 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 02:05:30.067649 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 02:05:31.442034 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 02:05:31.924757 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 02:05:32.707583 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 02:05:33.842345 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 02:05:34.269037 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 02:05:34.735373 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:05:35.168982 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 02:05:36.360540 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 02:05:36.718650 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 02:05:37.537405 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 02:05:38.995617 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 02:05:39.398479 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 02:05:39.759361 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:05:40.286708 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 02:05:41.454303 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 02:05:41.598075 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 02:05:42.871440 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 02:05:44.137867 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 02:05:44.143620 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 02:05:44.894141 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:45.445477 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 02:05:46.364917 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 02:05:46.815503 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 02:05:48.079901 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 02:05:48.944255 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 02:05:49.309427 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 02:05:50.107093 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:50.695971 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 02:05:51.453227 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 02:05:51.454545 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 02:05:53.185900 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 02:05:53.967993 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 02:05:54.045145 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 02:05:55.174572 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:05:55.658308 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 02:05:56.438776 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 02:05:56.514393 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 02:05:58.313819 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 02:05:59.145484 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 02:05:59.337565 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 02:06:00.190599 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:06:00.887761 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 02:06:01.568468 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 02:06:01.572217 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 02:06:03.374349 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 02:06:03.904482 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 02:06:04.122442 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 02:06:05.191021 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:06:05.701427 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 02:06:06.347618 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 02:06:06.606992 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 02:06:08.236515 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 02:06:09.160307 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 02:06:09.194519 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 02:06:10.255139 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:06:10.681310 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 02:06:11.634984 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 02:06:11.650950 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 02:06:13.310382 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 02:06:13.999310 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 02:06:14.257347 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 02:06:15.466496 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:06:15.904603 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 02:06:16.524700 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 02:06:17.023540 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 02:06:18.589067 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 02:06:18.955066 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 02:06:19.547346 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 02:06:20.518004 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:06:21.014362 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 02:06:21.508281 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 02:06:22.047263 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 02:06:23.526827 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (420 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:06:23.552502 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 02:06:23.963737 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 02:06:24.250676 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (423 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:06:24.580566 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 02:06:25.628623 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:06:26.070829 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 02:06:26.135345 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 02:06:26.853303 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 02:06:28.572268 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 02:06:28.668311 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 02:06:29.285138 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:06:30.095978 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:06:30.999141 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:06:30.999889 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 02:06:31.084699 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 02:06:31.435278 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 02:06:33.457317 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 02:06:33.761863 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 02:06:33.998417 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 02:06:35.953136 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 02:06:36.160758 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:06:36.357630 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 02:06:36.700407 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 02:06:38.209297 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 02:06:38.768851 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 02:06:39.055115 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 02:06:40.611964 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 02:06:41.283962 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:06:41.399346 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 02:06:41.716840 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 02:06:42.811734 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 02:06:43.830073 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 02:06:44.234781 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 02:06:45.648164 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 02:06:46.469734 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:06:46.470641 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 02:06:46.638852 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 02:06:48.091073 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 02:06:49.120067 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 02:06:49.233970 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 02:06:50.595150 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 02:06:51.508306 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:06:51.509126 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 02:06:51.664381 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 02:06:53.006767 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 02:06:53.933413 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 02:06:54.162210 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 02:06:55.530053 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 02:06:56.186772 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 02:06:56.573171 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:06:56.573959 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 02:06:57.798722 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 02:06:58.566606 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 02:06:58.882216 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 02:07:00.261974 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 02:07:00.945382 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 02:07:01.477793 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 02:07:01.874167 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:07:02.787415 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 02:07:03.289749 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 02:07:04.300077 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 02:07:05.189311 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 02:07:05.949442 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 02:07:06.737231 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 02:07:06.884613 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:07:07.699205 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 02:07:08.456724 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 02:07:09.514977 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 02:07:09.972815 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 02:07:10.981463 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 02:07:11.923913 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:07:12.080433 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 02:07:12.493304 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 02:07:13.844874 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 02:07:14.599922 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 02:07:15.162727 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 02:07:16.387374 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 02:07:17.187230 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:07:17.215481 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 02:07:17.780156 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 02:07:18.825101 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 02:07:19.765350 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 02:07:20.261444 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 02:07:21.352470 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 02:07:22.249795 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:07:22.250627 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 02:07:22.891247 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 02:07:23.609019 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (480 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:07:23.890461 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 02:07:24.326731 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (483 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:07:25.043241 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 02:07:25.401245 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 02:07:26.082850 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 02:07:27.325363 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:07:27.752700 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 02:07:28.049427 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 02:07:28.177789 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 02:07:30.244788 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 02:07:30.743412 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 02:07:30.863547 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:07:31.247071 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:07:32.360227 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:07:32.511742 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 02:07:33.291244 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 02:07:33.330566 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 02:07:34.517748 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 02:07:35.632704 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 02:07:35.815361 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 02:07:36.804162 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 02:07:37.394731 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:07:38.218117 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 02:07:38.524504 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 02:07:39.373844 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 02:07:40.632995 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 02:07:41.025527 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 02:07:41.960223 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 02:07:42.480549 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:07:43.284474 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 02:07:43.334153 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 02:07:44.444191 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 02:07:45.488755 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 02:07:45.760624 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 02:07:46.764333 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 02:07:47.716638 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:07:47.717479 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 02:07:48.371982 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 02:07:49.204790 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 02:07:50.313375 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 02:07:50.561009 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 02:07:51.863225 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 02:07:52.611420 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 02:07:52.722431 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:07:52.974423 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 02:07:54.088764 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 02:07:55.115793 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 02:07:55.436203 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 02:07:56.567975 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 02:07:57.390663 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 02:07:57.808144 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:07:58.051774 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 02:07:58.948534 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 02:07:59.843171 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 02:08:00.524799 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 02:08:01.598508 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 02:08:02.248764 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 02:08:03.059623 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:08:03.164288 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 02:08:04.058080 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 02:08:04.605177 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 02:08:05.661012 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 02:08:06.567024 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 02:08:07.159103 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 02:08:08.102172 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:08:08.102849 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 02:08:09.253990 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 02:08:09.438789 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 02:08:10.703783 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 02:08:11.652909 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 02:08:12.040681 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 02:08:13.195009 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:13.195847 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 02:08:14.222132 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 02:08:14.568080 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 02:08:15.808008 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 02:08:16.682848 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 02:08:17.135913 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 02:08:18.291888 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:18.504132 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 02:08:19.120131 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 02:08:19.703709 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 02:08:21.004637 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 02:08:21.844090 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 02:08:22.517200 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 02:08:23.325672 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:08:23.469138 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 02:08:23.692081 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (540 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:08:24.355199 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 02:08:24.401016 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (543 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:08:25.408697 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 02:08:26.084237 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 02:08:26.711469 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 02:08:28.129501 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 02:08:28.599225 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:28.662786 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 02:08:29.230291 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 02:08:30.629738 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 02:08:31.056376 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 02:08:31.603597 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:08:32.398298 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:08:33.162789 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 02:08:33.607036 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:08:33.698273 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 02:08:33.892179 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 02:08:35.535444 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 02:08:36.129428 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 02:08:36.350180 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 02:08:38.245016 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 02:08:38.541145 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 02:08:38.728361 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:08:38.729140 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 02:08:40.898429 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 02:08:41.141240 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 02:08:41.356668 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 02:08:43.463550 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 02:08:43.519639 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 02:08:43.872380 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:43.873240 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 02:08:45.938946 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 02:08:46.292742 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 02:08:46.504554 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 02:08:48.608125 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 02:08:48.658617 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 02:08:49.020654 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:49.021951 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 02:08:51.294837 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 02:08:51.351078 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 02:08:51.513829 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 02:08:53.809460 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 02:08:53.853758 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 02:08:54.065374 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:54.066232 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 02:08:56.052753 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 02:08:56.380811 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 02:08:56.461169 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 02:08:58.605716 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 02:08:58.749253 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 02:08:59.154083 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:08:59.154980 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 02:09:01.122013 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 02:09:01.160288 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 02:09:01.393976 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 02:09:03.628757 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 02:09:03.866214 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 02:09:03.999790 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 02:09:04.464134 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:09:06.185786 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 02:09:06.359843 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 02:09:06.438141 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 02:09:08.743744 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 02:09:08.759902 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 02:09:08.916311 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 02:09:09.606505 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:09:10.925840 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 02:09:11.221580 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 02:09:11.280435 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 02:09:13.380684 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 02:09:13.396723 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 02:09:13.629325 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 02:09:14.622216 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:09:15.647710 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 02:09:15.886768 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 02:09:16.128583 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 02:09:18.338671 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 02:09:18.374573 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 02:09:18.414829 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 02:09:19.723173 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:09:20.850237 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 02:09:20.874567 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 02:09:20.933215 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 02:09:23.207350 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 02:09:23.299477 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 02:09:23.776237 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (600 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:09:23.812368 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 02:09:24.478754 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (603 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:09:24.797848 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:09:25.688126 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 02:09:25.899235 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 02:09:26.299003 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 02:09:27.990588 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 02:09:28.567768 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 02:09:29.008236 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 02:09:29.937150 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:09:30.370455 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 02:09:30.937071 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 02:09:31.301118 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 02:09:32.662906 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 02:09:33.516809 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:09:33.544400 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:09:33.906898 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 02:09:35.053318 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:09:35.054272 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 02:09:36.142979 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 02:09:36.314802 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 02:09:37.505883 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 02:09:38.751017 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 02:09:39.040872 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 02:09:39.835377 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 02:09:40.100410 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:09:41.486119 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 02:09:41.780440 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 02:09:42.439692 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 02:09:43.817221 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 02:09:44.088110 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 02:09:45.023780 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 02:09:45.411796 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:09:46.621649 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 02:09:46.724869 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 02:09:47.426191 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 02:09:49.097152 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 02:09:49.188339 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 02:09:49.894186 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 02:09:50.458683 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:09:51.696456 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 02:09:51.793967 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 02:09:52.273049 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 02:09:54.185494 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 02:09:54.453965 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 02:09:54.542331 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 02:09:55.852438 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:09:56.446561 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 02:09:56.671408 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 02:09:57.063624 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 02:09:58.758304 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 02:09:59.338855 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 02:09:59.901003 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 02:10:00.897490 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:10:01.442688 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 02:10:01.720254 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 02:10:02.268730 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 02:10:04.123953 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 02:10:04.208833 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 02:10:04.720952 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 02:10:06.128539 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:06.505828 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 02:10:06.747241 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 02:10:07.281577 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 02:10:08.987377 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 02:10:09.145069 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 02:10:09.478127 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 02:10:11.130879 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:10:11.548835 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 02:10:11.611616 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 02:10:12.225912 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 02:10:13.885147 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 02:10:14.131943 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 02:10:14.776216 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 02:10:16.187661 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:10:16.388074 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 02:10:16.762965 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 02:10:17.086755 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 02:10:18.789183 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 02:10:19.425639 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 02:10:19.883298 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 02:10:21.480474 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:10:21.481365 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 02:10:22.071864 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 02:10:22.629739 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 02:10:23.862864 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (660 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:10:24.056895 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 02:10:24.552117 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (663 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:10:24.744091 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 02:10:24.956003 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 02:10:26.620997 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:26.621997 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 02:10:27.221369 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 02:10:27.489202 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 02:10:29.295447 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 02:10:29.838559 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 02:10:29.929548 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 02:10:31.645731 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:31.646584 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 02:10:31.888667 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 02:10:32.628881 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 02:10:34.064390 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 02:10:34.215857 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:10:34.689141 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:10:34.983932 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 02:10:36.591544 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 02:10:36.836668 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:36.837497 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 02:10:37.444761 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 02:10:38.931942 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 02:10:39.428802 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 02:10:39.687588 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 02:10:41.623780 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 02:10:42.136879 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:42.137873 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 02:10:42.227238 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 02:10:44.288848 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 02:10:44.583714 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 02:10:44.685865 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 02:10:46.848904 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 02:10:46.868105 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 02:10:46.963505 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 02:10:47.292937 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:10:49.255625 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 02:10:49.265864 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 02:10:49.307204 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 02:10:51.655508 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 02:10:51.827180 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 02:10:51.927625 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 02:10:52.373100 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:10:54.186972 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 02:10:54.311266 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 02:10:54.734175 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 02:10:56.396250 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 02:10:56.670374 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 02:10:57.108235 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 02:10:57.626187 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:10:58.987519 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 02:10:59.440346 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 02:10:59.838654 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 02:11:01.467949 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 02:11:01.942512 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 02:11:02.182986 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 02:11:02.885113 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:11:04.056458 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 02:11:04.347378 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 02:11:04.526410 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 02:11:06.539081 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 02:11:06.895417 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 02:11:06.946393 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 02:11:07.939551 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:11:08.871879 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 02:11:09.179922 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 02:11:09.309417 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 02:11:11.567440 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 02:11:11.646528 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 02:11:11.893662 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 02:11:13.158203 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:11:14.089466 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 02:11:14.183733 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 02:11:14.494963 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 02:11:16.622234 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 02:11:16.696553 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 02:11:16.892848 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 02:11:18.531179 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20260811 02:11:19.039088 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 02:11:19.113446 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 02:11:19.119212 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 02:11:21.405369 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 02:11:21.577931 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 02:11:21.587721 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 02:11:23.595463 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:11:23.683604 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 02:11:23.943565 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (720 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:11:24.116514 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 02:11:24.288975 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 02:11:24.627336 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (723 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:11:26.187886 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 02:11:26.769232 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 02:11:26.945884 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 02:11:28.500303 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 02:11:28.968158 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:11:29.414289 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 02:11:29.424372 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 02:11:31.260352 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 02:11:31.711957 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 02:11:31.940260 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 02:11:33.766964 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 02:11:34.164081 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:11:34.320557 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 02:11:34.655508 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.10.87.62:36291) | 127.10.87.62:36291 | UNAVAILABLE
Error from 127.10.87.62:36291: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
FAILED
I20260811 02:11:35.866585 10588 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.10.87.62:36291, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:11:36.543906 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 02:11:36.965099 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 02:11:37.074877 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 02:11:39.127382 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 02:11:39.192955 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:11:39.242878 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 02:11:39.816736 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 02:11:41.699520 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 02:11:41.776865 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 02:11:42.166594 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 02:11:44.289805 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:11:44.290903 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 02:11:44.354445 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 02:11:44.547588 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 02:11:46.856385 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 02:11:47.012746 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 02:11:47.304307 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 02:11:49.496485 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20260811 02:11:49.588304 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 02:11:49.642062 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 02:11:49.929399 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 02:11:52.208884 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20260811 02:11:52.299242 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 02:11:52.430452 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
[New LWP 10591]
[New LWP 10592]
[New LWP 10594]
[New LWP 11126]
[New LWP 11127]
[New LWP 11128]
[New LWP 11129]
[New LWP 15591]
[New LWP 15592]
[New LWP 15593]
[New LWP 15594]
[New LWP 15598]
[New LWP 15599]
[New LWP 15600]
[New LWP 15601]
[New LWP 15602]
W20260811 02:11:54.784889 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:11:54.785725 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 02:11:54.849197 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 02:11:54.978348 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260811 02:11:57.378022 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 02:11:57.449088 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 02:11:57.459983 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 02:11:59.857264 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:11:59.858219 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 02:11:59.952555 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 02:12:00.191380 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 02:12:02.169601 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 02:12:02.407071 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 02:12:02.624675 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 02:12:04.704272 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 02:12:04.904197 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:12:04.904961 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 02:12:05.245807 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 02:12:06.928874 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 02:12:07.507696 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 02:12:07.798401 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 02:12:09.343477 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 02:12:09.670658 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 02:12:10.053848 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20260811 02:12:10.203967 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 02:12:12.081938 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 02:12:12.272683 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 02:12:12.478681 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 02:12:14.585281 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 02:12:14.731595 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 02:12:14.815786 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260811 02:12:15.062974 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:12:16.810665 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 02:12:17.210143 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260811 02:12:17.477361 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260811 02:12:19.222235 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
0x00007febd4358ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7febc763a280 (LWP 10588) "all_types-itest" 0x00007febd4358ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7febc5615700 (LWP 10591) "all_types-itest" 0x00007febd435d32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7febc3c3b700 (LWP 10592) "kernel-watcher-" 0x00007febd4358fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7febc343a700 (LWP 10594) "rpc reactor-105" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7febc2c39700 (LWP 11126) "rpc reactor-111" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7febc2231700 (LWP 11127) "rpc reactor-111" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7febc1a30700 (LWP 11128) "rpc reactor-111" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7febc122f700 (LWP 11129) "rpc reactor-111" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7febbfa2c700 (LWP 15591) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7febbd227700 (LWP 15592) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7febb9a20700 (LWP 15593) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7febc0a2e700 (LWP 15594) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7febbea2a700 (LWP 15598) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7febbe229700 (LWP 15599) "rpc reactor-155" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7febbda28700 (LWP 15600) "rpc reactor-156" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7febbca26700 (LWP 15601) "rpc reactor-156" 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7febbc225700 (LWP 15602) "client-negotiat" 0x00007febd4358fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7febbc225700 (LWP 15602)):
#0 0x00007febd4358fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260811 02:12:19.877239 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#1 0x00007febd3b3021c in kudu::ConditionVariable::WaitFor (this=0x7febbc224690, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:121
#2 0x00007febd3c5b23d in kudu::ThreadPool::DispatchThread (this=0x565464a20b40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:677
#3 0x00007febd3c5c08f in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x565464a08ef8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/threadpool.cc:815
#4 0x00007febd3c5d7db in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
W20260811 02:12:20.090405 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20260811 02:12:20.095918 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#5 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a08ef8) at /usr/include/c++/7/bits/std_function.h:706
#6 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a08ea0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#7 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7febbca26700 (LWP 15601)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465152b00, timeout=0.098919219570234418) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465152b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464a27a98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464a27a90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a096d8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a096d8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a09680) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7febbda28700 (LWP 15600)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465153b80, timeout=0.051200114656239748) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465153b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464a26e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464a26e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a09d98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a09d98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a09d40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7febbe229700 (LWP 15599)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465153600, timeout=0.098885331535711884) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465153600, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x5654649df698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x5654649df690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a09a38) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a09a38) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a099e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7febbea2a700 (LWP 15598)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465152000, timeout=0.098859055666252971) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465152000, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464a05298, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464a05290) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a09c78) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a09c78) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a09c20) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7febc0a2e700 (LWP 15594)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465154100, timeout=0.099070676835253835) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465154100, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464de5e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464de5e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464931a38) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464931a38) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x5654649319e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7febb9a20700 (LWP 15593)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x565465155700, timeout=0.099043773720040917) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x565465155700, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464de4698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464de4690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464931378) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464931378) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464931320) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7febbd227700 (LWP 15592)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c4c00, timeout=0.099041778827086091) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c4c00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464de5498, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464de5490) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464930178) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464930178) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464930120) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7febbfa2c700 (LWP 15591)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c5700, timeout=0.098965106764808297) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c5700, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464a05698, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464a05690) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464930b98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464930b98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464930b40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7febc122f700 (LWP 11129)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c4100, timeout=0.099279301008209586) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c4100, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x5654649dec98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x5654649dec90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a08838) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a08838) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a087e0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7febc1a30700 (LWP 11128)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c3b80, timeout=0.099268494872376323) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c3b80, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x5654649de498, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x5654649de490) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a08718) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a08718) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a086c0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7febc2231700 (LWP 11127)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c2b00, timeout=0.099230801919475198) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c2b00, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x5654649de898, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x5654649de890) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a085f8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a085f8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a085a0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7febc2c39700 (LWP 11126)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c2580, timeout=0.099200242897495627) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c2580, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464813c98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464813c90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x565464a084d8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x565464a084d8) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x565464a08480) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7febc343a700 (LWP 10594)):
#0 0x00007febd2243947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007febcdbcd25d in epoll_poll (loop=0x5654649c2000, timeout=0.099481129785999656) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007febcdbd1ba3 in ev_run (loop=0x5654649c2000, flags=0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007febd3c4480f in ev::loop_ref::run (this=0x565464812e98, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007febcfd030f1 in kudu::rpc::ReactorThread::RunThread (this=0x565464812e90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:611
#5 0x00007febcfcffbcf in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x5654648e7d98) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/reactor.cc:299
#6 0x00007febcfd078d1 in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x5654648e7d98) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x5654648e7d40) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7febc3c3b700 (LWP 10592)):
#0 0x00007febd4358fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007febd3b30033 in kudu::ConditionVariable::WaitUntil (this=0x5654648e7f38, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:87
#2 0x00007febd3bb12f4 in kudu::CountDownLatch::WaitUntil (this=0x5654648e7f00, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:89
#3 0x00007febd3bb139e in kudu::CountDownLatch::WaitFor (this=0x5654648e7f00, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:99
#4 0x00007febd3bb03b2 in kudu::KernelStackWatchdog::RunThread (this=0x5654648e7e60) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007febd3baf5c1 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x5654648e7b58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007febd3bb0d16 in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
#7 0x00005654494f035a in std::function<void ()>::operator()() const (this=0x5654648e7b58) at /usr/include/c++/7/bits/std_function.h:706
#8 0x00007febd3c4bb52 in kudu::Thread::SuperviseThread (arg=0x5654648e7b00) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:694
#9 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7febc5615700 (LWP 10591)):
#0 0x00007febd435d32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007febd3c423a1 in kudu::Subprocess::DoWait (this=0x7febc5614350, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:849
#2 0x00007febd3c3f73a in kudu::Subprocess::Wait (this=0x7febc5614350, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/subprocess.cc:561
#3 0x00007febd3ba4729 in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:260
#4 0x00007febd3ba4259 in kudu::PstackWatcher::RunGdbStackDump (pid=10588, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:218
#5 0x00007febd3ba3ab7 in kudu::PstackWatcher::DumpPidStacks (pid=10588, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:169
#6 0x00007febd3ba3a30 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/pstack_watcher.cc:162
#7 0x00005654494eee09 in kudu::<lambda()>::operator()(void) const (__closure=0x56546488bff8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:66
#8 0x00005654494ef60a in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9 0x00005654494ef35e in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x00005654494efd7e in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x56546488bff8) at /usr/include/c++/7/thread:234
#11 0x00005654494efd23 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x56546488bff8) at /usr/include/c++/7/thread:243
#12 0x00005654494efcde in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x56546488bff0) at /usr/include/c++/7/thread:186
#13 0x00007febd2b866df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007febd43526db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007febd224361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7febc763a280 (LWP 10588)):
#0 0x00007febd4358ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007febd3b2fead in kudu::ConditionVariable::Wait (this=0x565464989a90) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/condition_variable.cc:55
#2 0x00007febd4db3aa1 in kudu::CountDownLatch::Wait (this=0x565464989a58) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/countdown_latch.h:79
W20260811 02:12:21.627847 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#3 0x00007febd4b72414 in kudu::Synchronizer::Wait (this=0x7ffd706cdf60) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/async_util.h:58
W20260811 02:12:22.447926 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
W20260811 02:12:22.505488 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#4 0x00007febd4944c0a in kudu::client::KuduClient::Data::ConnectToCluster (this=0x5654648bcc80, client=0x565465116f80, deadline=..., creds_policy=kudu::rpc::CredentialsPolicy::ANY_CREDENTIALS) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client-internal.cc:828
#5 0x00007febd49068a5 in kudu::client::KuduClientBuilder::Build (this=0x7ffd706ce3a0, client=0x565464930470) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/client.cc:474
#6 0x00007febccfafa93 in kudu::tools::CreateKuduClient (master_addresses=std::vector of length 1, capacity 1 = {...}, client=0x565464930470, can_see_all_replicas=true) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/tool_action_common.cc:773
W20260811 02:12:24.028684 10978 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (780 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
W20260811 02:12:24.077740 10710 consensus_peers.cc:597] T c7c13910df10490f88a4decb1d62b616 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#7 0x00007febd202fac4 in kudu::tools::RemoteKsckCluster::Connect (this=0x565464930360) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck_remote.cc:530
#8 0x00007febd1fe788e in kudu::tools::Ksck::CheckClusterRunning (this=0x56546516d410) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:459
#9 0x00007febd1fe8e84 in kudu::tools::Ksck::Run (this=0x56546516d410) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:594
#10 0x00007febd1feb27b in kudu::tools::Ksck::RunAndPrintResults (this=0x56546516d410) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/ksck.cc:800
#11 0x00007febd4d84044 in kudu::ClusterVerifier::RunKsck (this=0x7ffd706ce8b0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:113
W20260811 02:12:24.703369 10833 heartbeater.cc:646] Failed to heartbeat to 127.10.87.62:36291 (783 consecutive failures): Network error: Failed to ping master at 127.10.87.62:36291: Client connection negotiation failed: client connection to 127.10.87.62:36291: connect: Connection refused (error 111)
#12 0x00007febd4d83a03 in kudu::ClusterVerifier::CheckCluster (this=0x7ffd706ce8b0) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x00005654494a7227 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x5654647df900, scanner_setup=..., verifier=...) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:622
W20260811 02:12:24.974428 10710 consensus_peers.cc:597] T 0c339411098847e897c41c9b14d76167 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1806: this message will repeat every 5th retry.
W20260811 02:12:25.113322 10710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20260811 02:12:25.114230 10710 consensus_peers.cc:597] T b0de48471b894bb8bbfaeca77de15585 P dd70368ef7ad4752afbe694f0aef10b7 -> Peer f4989bac1fb24983965dc41765af59fc (127.10.87.3:39003): Couldn't send request to peer f4989bac1fb24983965dc41765af59fc. Status: Network error: Client connection negotiation failed: client connection to 127.10.87.3:39003: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#14 0x000056544947a803 in kudu::client::AllTypesItest_TestAllKeyTypes_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::TestBody (this=0x5654647df900) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/all_types-itest.cc:676
#15 0x00007febd3d7d0ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7febd3d7e82b "the test body", method=<optimized out>, object=0x5654647df900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x5654647df900, method=<optimized out>, location=location@entry=0x7febd3d7e82b "the test body") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007febd3d71bdb in testing::Test::Run (this=0x5654647df900) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 0x00007febd3d71d9d in testing::TestInfo::Run (this=0x5654647f46c0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#19 0x00007febd3d72377 in testing::TestSuite::Run (this=0x5654647f47e0) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#20 0x00007febd3d7277c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#21 0x00007febd3d7d60d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7febd3d803c8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x5654648b6280) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#22 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5654648b6280, method=<optimized out>, location=location@entry=0x7febd3d803c8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#23 0x00007febd3d71e63 in testing::UnitTest::Run (this=0x7febd3f911e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#24 0x00005654494f00d2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#25 0x00005654494ef2bf in main (argc=1, argv=0x7ffd706ceeb8) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_main.cc:115
************************* END STACKS ***************************
F20260811 02:12:25.318640 10591 test_main.cc:69] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786414345 (unix time) try "date -d @1786414345" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000295c) received by PID 10588 (TID 0x7febc5615700) from PID 10588; stack trace: ***
@ 0x7febd435d980 (unknown) at ??:0
@ 0x7febd2160e87 gsignal at ??:0
@ 0x7febd21627f1 abort at ??:0
@ 0x7febd2e65d5d google::LogMessage::Fail() at ??:0
@ 0x7febd2e69b23 google::LogMessage::SendToLog() at ??:0
@ 0x7febd2e6575c google::LogMessage::Flush() at ??:0
@ 0x7febd2e66ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5654494eef46 _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at ??:0
@ 0x5654494ef60a _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x5654494ef35e _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at ??:0
@ 0x5654494efd7e _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x5654494efd23 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at ??:0
@ 0x5654494efcde _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at ??:0
@ 0x7febd2b866df (unknown) at ??:0
@ 0x7febd43526db start_thread at ??:0
@ 0x7febd224361f clone at ??:0